Method and system for display of an electronic representation of physical effects and property damage resulting from a parametric natural disaster event

ABSTRACT

A method is provided for displaying an augmented reality (AR) representation of physical effects and property damage resulting from a parametric earthquake event. The method includes scanning, using one or more sensors of a user device, a scene in proximity to a user. The method also includes identifying a background and objects in the scene. The method further includes creating an AR background for the background of the scene and AR objects for the objects in the scene. In addition, the method includes receiving at least one seismic characteristic from the user through the display of the user device. The method additionally includes displaying at least one seismic effect on the AR objects and the AR background in the scene based on the at least one received seismic characteristic.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 63/152,739 filed on Feb. 23, 2021 and entitled METHOD AND SYSTEM FORDISPLAY OF AN ELECTRONIC REPRESENTATION OF PHYSICAL EFFECTS AND PROPERTYDAMAGE RESULTING FROM A PARAMETRIC EARTHQUAKE EVENT. This applicationalso claims the benefit of U.S. Provisional Application No. 63/169,801,filed on Apr. 1, 2021, entitled METHOD AND SYSTEM FOR DISPLAY OF ANELECTRONIC REPRESENTATION OF PHYSICAL EFFECTS AND PROPERTY DAMAGERESULTING FROM A PARAMETRIC NATURAL DISASTER EVENT. U.S. ProvisionalApplication Ser. No. 63/152,739 and U.S. Provisional Application No.63/169,801 are incorporated by reference in their entireties.

TECHNICAL FIELD

The present invention relates to techniques and technology for producingan electronic representation of the effects of a parametric earthquakeevent on a specific property comprising one or more structures, and morespecifically to methods and systems of processing video and still imagesof the specific property taken during normal conditions to show thecalculated movement and damage to the respective structures of theproperty attributable to a parametric earthquake event.

BACKGROUND

The public is familiar with fictious depictions of earthquakes frommotion pictures and even the depiction of actual earthquake events indocumentary footage. However, such depictions do not serve to inform thepublic regarding the likely damage to their own property (both real andpersonal property) in the event of an earthquake event. Betterinformation regarding the effects of earthquake events on property at aspecific geographic location and the likely property damage resultingfrom such earthquake events can be useful in assessing the need forearthquake preparation.

A need therefore exists, for a system that can identify structurescomprising property pictured at a specific geographic location anddisplay an electronic representation, e.g., a video program and/oraudiovisual program, showing a representation of the motion of therespective structures caused by a seismic event having known parameters,e.g., location, strength and/or duration.

A need further exists, for a system that can identify structurescomprising a property pictured at a specific geographic location anddisplay an electronic representation, e.g., still photo images, a videoprogram and/or an audiovisual program showing a representation of damageto the respective structures caused by a seismic event having knownparameters, e.g., location, strength and/or duration.

SUMMARY

In one aspect, a system receives images of property at a specifiedgeographic location. The system processes, using an image recognitionprocessor, the received images and identifies one or more structurescomprising the property. The system retrieves parametric data regardinga designated seismic event. The system defines a key data paircorresponding to the specified geographic location and the designatedseismic event. The system determines key attributes relating the keydata pair. The system determines, using a computer or processor, valuesof a deemed seismic action at the specified geographic location bymodifying the parametric data for the designated seismic event using thekey attributes. The system produces images showing an electronicrepresentation the one or more structures comprising the property movingin accordance with the deemed seismic action.

In one embodiment thereof, the images comprise a video program showingan electronic representation of the one or more structures moving inaccordance with the deemed seismic action.

In another embodiment thereof, the images comprise an audiovisualprogram showing an electronic representation of the one or morestructures moving in accordance with the deemed seismic action.

In yet another embodiment thereof, a key attribute determined by thesystem is the distance from the designated seismic event to thespecified geographic location.

In still another embodiment thereof, a key attribute determined by thesystem is the intervening geology between the designated seismic eventto the specified geographic location.

In a further embodiment thereof, the system assigns to one or more ofthe identified structures a respective structure attribute. The systemdetermines values of deemed structure action for each respectivestructure by modifying the values of the deemed seismic action at thespecified geographic location using the respective structure attribute.The system produces images showing an electronic representation of theone or more structures moving in accordance with the deemed seismicaction and as further modified by the deemed structure action.

In a yet further embodiment thereof, the respective structure attributeis correlated to a determined mass of the structure.

In a still further embodiment thereof, the respective structureattribute is correlated to a determined resonant frequency of thestructure.

In another aspect, a system receives images of property at a specifiedgeographic location. The system processes, using an image recognitionprocessor, the received images and identifies one or more structurescomprising the property. The system receives parametric data regarding adesignated seismic event. The system defines a key data paircorresponding to the specified geographic location and the designatedseismic event. The system determines key attributes relating the keydata pair. The system determines, using a computer or processor, valuesof a deemed seismic action at the specified geographic location bymodifying the parametric data for the designated seismic event using thekey attributes. The system determines a respective deemed damagecorresponding to each structure based on the deemed seismic action. Thesystem produces images showing an electronic representation the one ormore structures comprising the property modified with respectiveelectronic representations of the respective deemed damage.

In one embodiment thereof, the images comprise a video program showingan electronic representation of the one or more structures modified toshow the respective deemed damage.

In another embodiment thereof, the images comprise an audiovisualprogram showing an electronic representation of the one or morestructures modified to show the respective deemed damage.

In yet embodiment thereof, the images comprise one or more still imagesshowing an electronic representation of the one or more structuresmodified to show the respective deemed damage.

In still another embodiment thereof, a key attribute determined by thesystem is the distance from the designated seismic event to thespecified geographic location.

In a further embodiment thereof, a key attribute determined by thesystem is the intervening geology between the designated seismic eventto the specified geographic location.

In a yet further embodiment thereof, the system assigns to one or moreof the identified structures a respective structure attribute. Thesystem determines values of deemed structure damage for each respectivestructure by modifying the values of the deemed seismic action at thespecified geographic location using the respective structure attribute.The system produces images showing an electronic representation of theone or more structures modified with the respective deemed damagecorresponding to each structure based on the deemed seismic action andfurther corresponding to the respective structure attribute.

In a still further embodiment thereof, the respective structureattribute is correlated to a determined mass of the structure.

In another embodiment thereof, the respective structure attribute iscorrelated to a determined resonant frequency of the structure.

In yet another embodiment thereof, the respective structure attribute iscorrelated to a construction material of the structure.

In a third aspect, a method is provided for displaying an augmentedreality (AR) representation of physical effects and property damageresulting from a parametric earthquake event. The method includesscanning, using one or more sensors of a user device, a scene inproximity to a user. The method also includes identifying a backgroundand objects in the scene. The method further includes creating an ARbackground for the background of the scene and AR objects for theobjects in the scene. The method further includes displaying the ARbackground and the AR objects on the display of the user device. Inaddition, the method includes receiving at least one seismiccharacteristic from the user through the display of the user device. Themethod additionally includes displaying at least one seismic effect onthe AR objects and on the AR background in the scene displayed on theuser device based on the at least one received seismic characteristic.

In one embodiment thereof, the background is a wall and the objects arepicture frames.

In another embodiment thereof, displaying the at least one seismiceffect on the AR objects comprises independently applying the at leastone seismic effect on the AR objects and the AR background.

In another embodiment thereof, displaying the at least one seismiceffect on the AR objects comprises independently applying the at leastone seismic effect using a first effect type on the AR objects and usinga second effect type on the AR background.

In yet another embodiment thereof, the first effect type is applying anAR oscillation and the second effect type is generating an AR crack.

In another embodiment thereof, independently applying the at least oneseismic effect using a first effect type on the AR objects and using asecond effect type on the AR background comprises selecting the firsteffect type based on a first identity of the AR objects and selectingthe second effect type based on a second identity of the AR background.

In yet embodiment thereof, displaying the at least one seismic effect onthe AR background comprises generating an AR crack on the AR background.

In yet another embodiment thereof, displaying the at least one seismiceffect on the AR objects comprises generating an AR crack on at leastone of the AR objects.

In yet another embodiment thereof, displaying the at least one seismiceffect on the AR objects comprises applying an AR oscillation on atleast one of the AR objects.

In still another embodiment thereof, displaying the at least one seismiceffect on the AR objects comprises breaking an AR object into two ormore AR partial objects, wherein each of the AR partial objects issmaller than the AR object.

In still another embodiment thereof, breaking the AR object into two ormore AR partial objects comprises selecting respective sizes for the ARpartial objects such that a cumulative size of the two or more partialobjects is equal to a size of the AR object.

In a further embodiment thereof, displaying the at least one seismiceffect on the AR objects comprises generating an AR crack on the ARbackground.

In a further embodiment thereof, generating an AR crack on the ARbackground comprises adding dynamic shading to the AR crack simulatingparallax effects based on point of view.

In a yet further embodiment thereof, the method further includesdisplaying a control for a seismic intensity as one of the at least oneseismic characteristic.

In a fourth aspect, a non-transitory computer readable medium isprovided for displaying an augmented reality (AR) representation ofphysical effects and property damage resulting from a parametricearthquake event. The non-transitory computer readable medium containsinstructions that when executed cause a processor to scan, using one ormore sensors of a user device, a scene in proximity to a user. Thenon-transitory computer readable medium also contains instructions thatwhen executed cause a processor to identify a background and objects inthe scene. The non-transitory computer readable medium further containsinstructions that when executed cause a processor to create an ARbackground for the background of the scene and AR objects for theobjects in the scene. The non-transitory computer readable mediumfurther contains instructions that when executed cause a processor todisplay the AR background and the AR objects on the display of the userdevice In addition, non-transitory computer readable medium containsinstructions that when executed cause a processor to receive at leastone seismic characteristic from the user through the display of the userdevice. The non-transitory computer readable medium additionallycontains instructions that when executed cause a processor to display atleast one seismic effect on the AR objects and on the AR background inthe scene displayed on the user device based on the at least onereceived seismic characteristic.

In one embodiment thereof, the background is a wall and the objects arepicture frames.

In another embodiment thereof, the instructions when executed cause theprocessor to display the at least one seismic effect on the AR objectscomprise instructions that when executed cause the processor toindependently apply the at least one seismic effect on the AR objectsand the AR background.

In yet embodiment thereof, the instructions when executed cause theprocessor to display the at least one seismic effect on the AR objectscomprise instructions that when executed cause the processor to generatea crack on at least one of the AR objects.

In still another embodiment thereof, the instructions when executedcause the processor to display the at least one seismic effect on the ARobjects comprise instructions that when executed cause the processor tobreak an AR object into two or more AR partial objects. A cumulativesize of the two or more partial objects is equal to a size of the ARobject.

In a further embodiment thereof, the instructions when executed causethe processor to display the at least one seismic effect on the ARobjects comprise instructions that when executed cause the processor togenerate a crack on the AR background.

In a yet further embodiment thereof, the instructions when executedfurther cause the processor to display a control for a seismic intensityas one of the at least one seismic characteristic.

In a fifth aspect, a user device is provided for displaying an augmentedreality (AR) representation of physical effects and property damageresulting from a parametric earthquake event. The user device includesone or more sensors, a display, and a processor operably couple to theone or more sensors and the display. The one or more sensors areconfigured to scan a scene in proximity to a user. The display isconfigured to display the scene. The processor is configured to identifya background and objects in the scene. The processor is also configuredto create an AR background for the background of the scene and ARobjects for the objects in the scene. The processor is furtherconfigured to display the AR background and the AR objects on thedisplay of the user device. The processor is further configured toreceive at least one seismic characteristic from the user through thedisplay of the user device. The processor is additionally configured todisplay at least one seismic effect on the AR objects and on the ARbackground in the scene displayed on the user device based on the atleast one received seismic characteristic.

In one embodiment thereof, the background is a wall and the objects arepicture frames.

In another embodiment thereof, the processor is further configured toindependently apply the at least one seismic effect on the AR objectsand the AR background.

In another embodiment thereof, the processor is further configured toapply the at least one seismic effect using a first effect type on theAR objects and using a second effect type on the AR background.

In yet another embodiment thereof, the first effect type is applying anAR oscillation and the second effect type is generating an AR crack.

In another embodiment thereof, independently applying the at least oneseismic effect using a first effect type on the AR objects and using asecond effect type on the AR background comprises selecting the firsteffect type based on a first identity of the AR objects and selectingthe second effect type based on a second identity of the AR background.

In yet another embodiment thereof, displaying the at least one seismiceffect on the AR background comprises generating an AR crack on the ARbackground.

In yet another embodiment thereof, to display the at least one seismiceffect on the AR objects, the processor is further configured togenerate an AR crack on at least one of the AR objects.

In still another embodiment thereof, to display the at least one seismiceffect on the AR objects, the processor is further configured to breakan AR object into two or more AR partial objects, wherein each of the ARpartial objects is smaller than the AR object.

In still another embodiment thereof, to display breaking the AR objectinto two or more AR partial objects, the processor is configured toselect respective sizes for the AR partial objects such that acumulative size of the two or more partial objects is equal to a size ofthe AR object.

In a further embodiment thereof, to display the at least one seismiceffect on the AR objects, the processor is further configured togenerate an AR crack on the AR background.

In a further embodiment thereof, to generate an AR crack on the ARbackground, the processor is further configured to add dynamic shadingto the AR crack simulating parallax effects based on point of view.

In a yet further embodiment thereof, the processor is further configuredto display a control for a seismic intensity as one of the at least oneseismic characteristic.

In a sixth aspect, a method is provided for displaying an augmentedreality (AR) representation of physical effects and property damageresulting from a parametric earthquake event including a damageinventory and assessment. The method includes scanning, using one ormore sensors of a user device, a scene in proximity to a user. Themethod also includes identifying a background and objects in the scene.The method further includes determining a number of the objectsidentified in the scene. The method further includes determining an areaof the background identified in the scene. The method further includescreating an AR background for the background of the scene and AR objectsfor the objects in the scene. The method further includes displaying theAR background and the AR objects on the display of the user device. Inaddition, the method includes receiving at least one seismiccharacteristic from the user through the user device. The methodadditionally includes displaying at least one seismic effect on the ARobjects and on the AR background in the scene displayed on the userdevice based on the at least one received seismic characteristic. Themethod further includes determining a seismic damage factor based on theat least one received seismic characteristic. The method furtherincludes determining an estimated object damage amount for the objectsidentified in the scene based on the determined number of objects andthe determined seismic damage factor. The method further includesdetermining an estimated structure damage amount for the backgroundidentified in the scene based on the determined area of the backgroundand the determined seismic damage factor.

In one embodiment, determining an estimated object damage amount furtherincludes assigning a respective initial value to each respective objectidentified in the scene, determining a respective final value for eachobject based a respective assigned value and the respective determinedseismic damage factor, and totaling the respective differences betweenthe respective initial values and the respective final values for allthe respective objects.

In another embodiment, the method further includes displaying theestimated object damage amount to the user.

In yet another embodiment, determining an estimated structure damageamount further includes assigning an initial unit value to thebackground identified in the scene, determining a respective final unitvalue for the background based an assigned unit value and the determinedseismic damage factor, and multiplying the differences between theinitial unit value and the final unit value by the determined area ofthe background.

In still another embodiment, the method further includes displaying theestimated structure damage amount to the user.

In another embodiment, the method further includes categorizing eachidentified object into one of at least two predetermined object classes,wherein a first initial value is assigned to each of the objectscategorized in the first predetermined object class, and a secondinitial value is assigned to each of the objects categorized in thesecond predetermined object class.

In yet another embodiment, the method further includes displaying to theuser a list of the respective objects identified in the scene and therespective initial values, allowing the user to provide a respectivecustom value for each respective object, and replacing the respectiveinitial value with the respective custom value as the respectiveassigned value used to determine the respective final value of therespective object.

In still another embodiment, the method further includes displaying tothe user the initial unit value for the background, allowing the user toprovide a custom unit value for the background, and replacing theinitial unit value with the custom unit value as the assigned unit valueused to determine the final unit value for the estimated structuredamage.

In another embodiment, the method further includes creating a first listof objects identified in a first scene and their respective assignedvalues, creating a second list of objects identified in a second sceneand their respective assigned values, and combining the first list andthe second list into a consolidated object list including all the objectidentified in the first scene and in the second scene and theirrespective assigned values.

In yet another embodiment, the method further includes using theconsolidated object list to determine a consolidated object damage valuefor all the objects identified in the first scene and in the secondscene and displaying the consolidated object damage value to the user.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding, reference is now made to thefollowing description taken in conjunction with the accompanyingDrawings in which:

FIG. 1 is a block diagram of a system for display of an electronicrepresentation of physical effects and property damage resulting from aparametric earthquake event in accordance with this disclosure;

FIG. 2 is a map showing the position of a parametric earthquake eventand several geographic locations;

FIG. 3 shows an image of an actual property comprising multiplestructures;

FIG. 4 shows an image of an electronic representation of the multiplestructures of the property of FIG. 3 moving in accordance with thedeemed seismic action from a parametric earthquake event;

FIG. 5 shows an image of an electronic representation of the multiplestructures of the property of FIG. 3 modified with electronicrepresentations of deemed damage from a parametric earthquake event;

FIG. 6 shows an image of actual roofing of a structure;

FIG. 7 shows a first damage texture representing damaged roofing;

FIG. 8A shows a first texture map corresponding to a first amount ofdeemed damage, and FIG. 8B shows the first texture map applied to thefirst damage texture to produce an electronic representation of a firstlevel of deemed damage;

FIG. 9A shows a second texture map corresponding to a second amount ofdeemed damage, and FIG. 9B shows the second texture map applied to thefirst damage texture to produce an electronic representation of a secondlevel of deemed damage;

FIG. 10A shows the image of the structure roofing of FIG. 6 modifiedusing the electronic representation of the first level of deemed damageto produce an electronic representation of the structure damaged inaccordance with the first level of deemed damage;

FIG. 10B shows the image of the structure roofing of FIG. 6 modifiedusing the electronic representation of the second first level of deemeddamage to produce an electronic representation of the structure damagedin accordance with the second level of deemed damage;

FIG. 11 shows an image of an actual structure on a property;

FIG. 12 shows a wireframe representation of the structure of FIG. 11after recognition of the original image by the image recognitionprocessor and processing into a wireframe by the graphics modelingprocessor;

FIG. 13 shows a damaged wireframe representation produced by modifyingthe original wireframe representation of the structure of FIG. 11 inaccordance with an assigned level of deemed damage received from thesystem processor;

FIG. 14 shows an electronic representation of the structure of FIG. 11damaged in accordance with the deemed damage level by rendering textureon the damaged wireframe representation and modifying the originaltexture with an electronic representation of damage in the texture inaccordance with the assigned level of deemed damaged;

FIGS. 15A-15C are block diagrams showing a method for implementing, on auser device, a system for display of an electronic representation ofphysical effects and property damage resulting from a parametricearthquake event in accordance with this disclosure;

FIGS. 16A-16E are block diagrams showing a method for implementing, on asystem server, a system for display of an electronic representation ofphysical effects and property damage resulting from a parametricearthquake event in accordance with this disclosure;

FIG. 17 is an image of an interior room of a structure in accordancewith this disclosure;

FIGS. 18A-18E are user interfaces for display of an electronicrepresentation of physical effects and property damage resulting from aparametric natural disaster event in accordance with this disclosure;

FIG. 19 is a block diagram of a system for display of an electronicrepresentation of physical effects and property damage resulting from aparametric earthquake event in accordance with this disclosure;

FIGS. 20A and 20B illustrate an example method for an AR earthquakevisualization and assessment process in accordance with this disclosure;

FIGS. 21A and 21B illustrate an example method for a scene generationprocess in accordance with this disclosure;

FIG. 22 illustrates an example meshes for assigning to a plane inaccordance with this disclosure;

FIGS. 23A and 23B illustrate an example object extraction from mesh inaccordance with this disclosure;

FIG. 24 illustrates an example method for a crack creation algorithm inaccordance with this disclosure;

FIGS. 25A and 25B illustrate an example UV mapping for the crackcreation algorithm in accordance with this disclosure;

FIGS. 26A-26C illustrate an example Voronoi noise for the crack creationalgorithm in accordance with this disclosure;

FIG. 27 illustrates an example AR earthquake visualization andassessment system in accordance with this disclosure, namely, from theviewpoint of a user holding a mobile device (the user's hand is notshown) displaying an AR earthquake scene based on the actual room scenevisible behind the mobile device;

FIGS. 28A and 28B illustrate example inventory tables for objects andbackgrounds identified in the scene in accordance with this disclosure;and

FIG. 29 illustrates an example method for an AR earthquake visualizationand assessment process in accordance with this disclosure.

DETAILED DESCRIPTION

Referring now to the drawings, wherein like reference numbers are usedherein to designate like elements throughout, the various views andembodiments of method and systems for display of an electronicrepresentation of physical effects and property damage resulting from aparametric natural disaster event are illustrated and described, andother possible embodiments are described. The figures are notnecessarily drawn to scale, and in some instances the drawings have beenexaggerated and/or simplified in places for illustrative purposes only.One of ordinary skill in the art will appreciate the many possibleapplications and variations based on the following examples of possibleembodiments.

Referring to FIG. 1 , there is illustrated a system 100 for display ofan electronic representation of physical effects and property damageresulting from a parametric earthquake event in accordance with oneaspect. Unless otherwise noted, in this application the terms“parametric earthquake event” and “parametric seismic event” have thesame meaning. A parametric earthquake event is a set of parameters andvalues for defining an earthquake event in terms of scientificprinciples, e.g., physics and geological principles. Such parameters caninclude, but are not limited to, date, time, epicenter location,hypocenter location (i.e., focus), duration, magnitude, peak groundacceleration (“PGA”), and maximum shaking intensity (“SI”), maximumground amplitude, mean ground amplitude, shaking frequency, soil type,rock type and fault classification. A parametric earthquake event caninclude some or all parameter values measured during an actual, i.e.,“historical” earthquake event, but can also include some or allparameters values selected by a user. The system 100 includes a systemserver 102 or computing device. The system server 102 contains one ormore processors, RAM memory, storage units and communication interfacesfor sending and receiving data to and from other devices. The systemserver 102 may comprise a single machine or multiple machines, includingvirtual machines resident on “cloud servers.” The system server 102 isoperably connected to a communication network 104 and to one or moresubsystems, e.g., an image recognition processor 106, a graphicsanimation processor 108 and/or an image rendering engine 110. In someembodiments, the communication network 104 is the internet; however, thecommunication network 104 can be any type of network that allows thesystem server 102 and subsystems 106, 108 and 110 to communicate withone another and with a user device 114. In the embodiment illustrated inFIG. 1, the server 102 is directly connected to the subsystems 106, 108and 110; however, in other embodiments some or all of the subsystems maybe connected to the server via the communication network 104. In someembodiments, the subsystems 106, 108 and 110 can communicate directlywith one another for improved data transfer.

The image recognition processor 106 can include or incorporate an imagelabeling or annotation tool (i.e., “labeling tool”). The labeling toolcan be used to process and label the images for bounding box objectdetection and segmentation so that the image is readable by machines. Insome embodiments, the labeling tool can utilize human assistance and inother embodiments the labeling tool can operate solely with machinelearning or artificial intelligence processes. In some embodiments,different image labeling tools may be used for processing images ofdifferent image types (e.g., interior images, exterior images, etc.).Using the image labeling tools, the various objects in the providedimage (e.g., user image 300, exterior image 1100, interior image 1700)can be labeled for specific purposes. In some embodiments, labeledobjects can be selected for replacement by a computer graphic object(e.g., 2D sprite or 3D polygon), which can be moved on-screen andotherwise manipulated as a single entity, e.g., for purposes of eventanimation. In some embodiments, labeled objects may be classified intodifferent types or categories of objects for different purposes. Forexample, in some embodiments, labeled objects can be categorized forproperties relating to event animation, e.g., movable-type objects,bendable-type objects, breakable-type objects, waterproof-type objects,water damageable-type objects, etc. In other embodiments, labeledobjects can be categorized for properties relating to inventory ordamage assessment, e.g., table-type objects, chair-type objects,window-type objects, hanging art-type objects, TV/computer screen-typeobjects. Such classification may be performed by the image recognitionprocessor 106 or by another processor, e.g., the system processor 102 orgraphics animation processor 108. The image labeling tools can use knownor future-developed detection techniques for detection of the objectincluding, but not limited to, semantic, bounding box, key-point andcuboid techniques.

A seismic event database 112 is operably connected to the server 102,either via the communication network 104 or directly. The seismic eventdatabase 112 stores parametric seismic event data corresponding to oneor more earthquakes or other seismic events. The parametric seismicevent data can include, but is not limited to, values for the followingparameters: event name, event date, event epicenter location, eventfocus (i.e., hypocenter) location, event duration, event magnitude,event PGA, event maximum shaking intensity (“SI”), event maximum groundamplitude, event mean ground amplitude, and event shaking frequency foreach seismic event. The seismic events data in the seismic eventdatabase 112 may be actual historical earthquake data, “relocated”earthquake data (i.e., where the majority of the data corresponds to ahistorical earthquake, but the epicenter/hypocenter location is changedto a different location) or hypothetical earthquake data specified by auser or otherwise generated. In some embodiments, the seismic eventdatabase 112 can be located within the server 102 or one of thesubsystems. In some embodiments, the seismic event database 112 caninclude data and/or values from public or private earthquake reportingagencies, such as the U.S. Geological Survey.

User devices 114, 115 can connect to the system 100 through thecommunication network 104. The user devices 114, 115 can be mobiledevices such as mobile phones, tablets, laptop computers or they can bestationary devices such as desktop computers or smart appliancesincluding, but not limited to, smart televisions. In some embodiments,aspects of the current system 100 may include downloadable software or“apps” resident on the user devices 114, 115 and/or non-downloadablesoftware (e.g., “cloud based software”) that remains resident on theserver 102 or other elements of the system 100 and is accessed by theuser device 114 via a web browser or other network interface.

Using the user devices 114, 115, system users can upload images ofactual property via the network 104 to the server 102. The images ofactual property can be captured using a camera 116 on the first userdevice 114 or from images stored in memory 118, e.g., a computer memory,hard drive, flash drive or other data storage technology. The images canbe video programs, audiovisual programs and/or single or multiple stillphoto images. User devices 114 can also transmit additional informationto the server 102 regarding the images, including, but not limited to,the geographic location of the property in the images (either entered bythe user as text, captured via GPS or wireless location information onthe user device 114, or captured vis geotagging information on the imagefile), the address of the property in the images, the name and/or othercontact information of the user, a desired parametric seismic eventand/or desired seismic parameters to be used in creating a parametricseismic event. Desired seismic parameters can include strength of theseismic event, duration of the seismic event, etc. Each of the seismicparameters can be input on the user devices 114, 115 by any suitablemeans. For example, the seismic parameters can be input or selectedusing a predetermined list, a slider associated with different values ofthe seismic parameters, a knob, a number entry. The inputs can bephysical components on the user device 114, 115 or virtualrepresentations on a display 120 of the user device 114, 115.

The system 100 processes, using an image recognition processor 106, thereceived images and identifies one or more structures comprising theproperty. The system 100 retrieves parametric data regarding adesignated seismic event from the seismic event database 112. The system100 defines a key data pair corresponding to the specified geographiclocation and the designated seismic event. The system 100 determines keyattributes relating the key data pair. The key attributes correlate tohow the parametric seismic data changes for distant geographic locations(i.e., at a distance from the event). In some embodiments, the values ofkey attributes are determined based principles known in physics and/orgeology including, but not limited to principles of seismic attenuation,resonant vibration and/or soil behavior factors. In some embodiments,the values of key attributes are determined based on seismic attenuationfactors and the distance between the property and the event center. Insome embodiments, the values of key attributes are determined based onresonant vibration factors, the shaking frequency of the event, and themass and/or resonant frequency of structures. In some embodiments, thevalues of key attributes are determined based on soil behavior factorsand the soil at the property location, the event center, and/or atintervening geological features. Some key attributes may vary in direct(i.e., linear) proportion to the distance between the geographiclocation of the property/structure and the seismic event center, whereasother key attributes may vary according to reciprocal square ofdistance, logarithmic decay of distance or other mathematical functionsrelating to the distance. In some embodiments, the values of keyattributes are selected by a user. Other key attributes may varydepending on other factors such as intervening geological featuresrather than distance. The system 100 determines, using a computer suchas the system processor 102, values of a deemed seismic action at thespecified geographic location by modifying the parametric data for thedesignated seismic event using all relevant key attributes. In someembodiments, multiple sets of values of the key attributes arepredetermined. In this case, the system processor 102 prepare aplurality of videos each combination of key attributes for each value ofthe key attributes.

Referring now also to FIG. 2 , there is illustrated a map 200 showingthe position of the epicenter of an exemplary parametric earthquakeevent 202 (“Event Q”), a first exemplary geographic location 204(“Location A”) and a second exemplary geographic location 206 (“LocationB”). In the map 200, the epicenter of parametric Event Q is positionedon a geological fault line 208; however, in other embodiments thesubject event can be positioned at any location as dictated by theparametric data values. The system 100 defines a key data paircorresponding to each pair of one specified geographic location (i.e.,the location of the structures/property received) and one designatedseismic event. For example, Location A and Event Q form a first key datapair for showing deemed motion and/or deemed damage toproperties/structures at Location A due to Event Q, and Location B andEvent Q form a second key data pair for showing deemed motion and/ordeemed damage to properties/structures at Location B due to Event Q. Theparametric seismic data for the event 202 can be obtained from theseismic event database 112 and can include relevant parametric data forthe event including, but not limited to, event name, event date, eventfocus location, event duration, event magnitude, event PGA and event SI.

The system 100 can determine key attributes associated with each keydata pair. The key attributes can relate to factors that change theeffect of the respective parametric seismic event of the key data pairon the respective geographic location of the same key data pair. Asillustrated in FIG. 2 , the system 100, e.g., using the system processor102, can determine a respective distance between the parametric seismicevent 202 and each respective geographic location 204, 206 associatedwith respective received property images and assign a respective keyattribute based on the respective determined distance. For example, inFIG. 2 , the distance between the epicenter of Event Q and Location A isshown by line 210, while the distance between Event Q and Location B isshown by line 212. Thus, the system 100 may assign a first distance keyattribute to the first key data pair (A, Q) based on the distance 210,and assign a second distance key attribute to the second key data pair(B, Q) based on the distance 212. In some embodiments, the distance fromthe geographic locations 204, 206 to the focus (hypocenter) location ofthe event 202 can be used instead of the distance to the epicenterlocation. In some embodiments, the system 100 can detect that the pathbetween the position of the event 202 and the geographic location 204,206 of a property/structure crosses an intervening geological feature214 that can change the effects of the seismic event. In the embodimentof FIG. 2 , the path between the second key data pair (B, Q) crosses anintervening geological feature 214. The system 100 may assign a geologickey attribute to the second key data pair (B, Q) based on a distance ofthe intervening geological feature 214, whereas the first key data pair(A, Q) would not receive a geologic key attribute. In some embodiments,the system 100 can modify the parametric seismic data using all keyattributes associated with the key data pair to determine the deemedseismic action and/or the deemed damage on structures at the respectivegeographic locations 204, 206. For example, the system 100 in theexample of FIG. 2 can modify the parametric seismic data for event 202in accordance with both distance key attributes and interveninggeological feature key attributes (where applicable) to determine thedeemed seismic action and deemed damage on structures at the respectivegeographic locations 204 and 206.

The system 100 produces images showing an electronic representation(s)for the one or more structures comprising the property moving inaccordance with the deemed seismic action. In some embodiments, theelectronic representation(s) of the structure can be produced using agraphics animation processor 108 and an image rendering engine 110. Insome embodiments, the images comprise a video program showing anelectronic representation(s) of the one or more structures moving withthe deemed seismic action. In another embodiment, the images comprise anaudiovisual program showing an electronic representation of the one ormore structures moving with the deemed seismic action.

In some embodiments, the system 100 assigns to one or more of theidentified structures a respective structure attribute. In someembodiments, the values of structure attributes are determined basedprinciples known in physics and/or engineering including, but notlimited to principles of vibrational loads, fatigue failure, strength ofmaterials and construction methods and failure analysis. The system 100determines values of deemed structure action for each respectivestructure by modifying the values of the deemed seismic action at thespecified geographic location using the respective structure attribute.The system 100 produces images showing an electronic representation ofthe one or more structures moving in accordance with the deemed seismicaction and as further modified by the deemed structure action. In someembodiments, the structure attribute is correlated to a determined massof the structure. In other embodiments, the respective structureattribute is correlated to a determined resonant frequency of thestructure. In some embodiments, the structure attribute is correlatedthe construction material of the structure. In some embodiments, thestructure attribute is correlated the construction method type of thestructure.

Referring now to FIG. 3 , there is shown an exemplary image 300 of aproperty disposed at a geographic location, such as first geographiclocation 204 or second geographic location 206. The image 300 showsnumerous man-made structures including a house 302, a garage 304, adriveway 306, a pool 308, natural structures including a yard 310 andtrees 312, and personal property items including automobiles 314. A userof the system 100 takes the image 300 of the property and transmits itfrom a user device 114 to the system processor 102 via communicationnetwork 104. In some embodiments, the image 300 may be a single image.In other embodiments, the image 300 may comprise a plurality or seriesof images such as a video program. The system processor 102 receives theimage(s) 300 of the property and sends them an image recognitionprocessor 106. Using the image recognition processor 106, the receivedimage(s) 300 are processed to identify the one or more structurescomprising the property using image recognition technology and/orartificial intelligence or machine learning technology. The system 100can receive parametric data regarding a designated seismic event fromthe seismic event database 112. The system 100 can define a key datapair corresponding to the specified geographic location of the propertyshown in the image 300 and the designated seismic event. The system 100can determine, using the system processor 102, values of a deemedseismic action at the specified geographic location of the propertyshown in the image 300 by modifying the parametric data for thedesignated seismic event using the key attributes.

Referring now to FIG. 4 , in some embodiments the system 100 can produceimages 400 showing an electronic representation 402, 404, 406, 408, 410,412, and 414 of the one or more structures 302, 304, 306, 308, 310, 312and 314 comprising the property moving in accordance with the deemedseismic action at the specified geographic location. For purposes ofillustration, the representation of movement in images 400 is denoted bybroken lines adjacent each structure. For example, the house 302 andgarage 304 are displayed with horizontal oscillation 416 and thevertical oscillation 418 in accordance with their respective deemedseismic action. The trees 312 are shown oscillating with bending(pendulum) motion 420 in accordance with their respective deemed seismicaction. The pool 308 is shown with splashing and overflowing water 422in accordance with its respective deemed seismic action. The electronicrepresentations 402, 404, 406, 408, 410, 412, and 414 of the respectivemoving structures 302, 304, 306, 308, 310, 312, and 314 can be modeledby the graphics animation processor 108 using the respective deemedseismic action and rendered by the image rendering engine 110 based onthe respective structures identified in the original image 300.

After producing the image of the electronic representations 400, 402,404, 406, 408, 410, 412, and 414 of the one or more structures 302, 304,306, 308, 310, 312, and 314 comprising the property moving in accordancewith the deemed seismic action at the specified geographic location, thesystem 100 can transmit the images 400 back to the first user device 114or to another user device 115 designated by the original request. Theuser can then view the electronic representation 400 using a displaydevice 120.

In another aspect, a system 100 can receive images of property at aspecified geographic location, processes, using an image recognitionprocessor 106, the received images, identify one or more structurescomprising the property and receive parametric data regarding adesignated seismic event, all substantially as previously described. Thesystem 100 can further define a key data pair corresponding to thespecified geographic location and the designated seismic event,determine key attributes relating the key data pair, and determine,using a computer, e.g., processor 102, values of a deemed seismic actionat the specified geographic location by modifying the parametric datafor the designated seismic event using the key attributes, again, allsubstantially as previously described. The system 100 can furtherdetermine a respective deemed damage corresponding to each structurebased on the deemed seismic action. The system 100 produces imagesshowing an electronic representation the one or more structurescomprising the property modified with respective electronicrepresentations of the respective deemed damage.

In some embodiments, the structure attribute used to determine thedeemed damage is correlated to a determined mass of the structure. Inother embodiments, the structure attribute is correlated to a determinedresonant frequency of the structure. In still other embodiments, thestructure attribute is correlated to a construction material of thestructure. Thus, structures built with shake-resistant materials orbuilding methods would have a different structure attribute thanstructures built with shake-damage prone materials or building methods.

Referring now to FIG. 5 , in one example, the system 100 can produceimages 500 showing an electronic representation 400, 402, 404, 406, 408,410, 412, and 414 the one or more structures 302, 304, 306, 308, 310,312 and 314 comprising the property modified with respective electronicrepresentations of the respective deemed damage. For example, theoriginal appearance of the house 302 from image 300 is modified withelectronic representations of cracked windows 502, wall cracks 504 andstructural damage 506. The original appearance of the garage 304 fromimage 300 is modified with electronic representations of structuralcollapse 508 and damaged automobiles. The original appearance of thedriveway 306 from image 300 is modified with electronic representationsof cracking 509. The original appearance of the pool 308 from image 300is modified with electronic representations of cracks 510 on the shelland decking. The original appearance of the trees 312 from image 300 ismodified with electronic repositioning to represent leaning or toppled512. The original appearance of the yard 310 from image 300 is modifiedwith electronic representations of washouts 514 near the pool 308 andlandslide/subsidence 516. cracks 510 on the shell and decking.

After producing image 500 the electronic representation 400, 402, 404,406, 408, 410, 412, and 414 of the one or more structures comprising theproperty modified to show the respective deemed damage associated with aparametric seismic event, the system 100 can transmit the images 500back to the original user device 114′ or to another user device 114″designated by the original request. The user can then view theelectronic representation 400 using a display device 120.

The system processor 102 can include any suitable hardware processor,such as a microprocessor, and in some embodiments, the hardwareprocessor can be controlled by a program stored in the memory and/orstorage. The image recognition processor 106, graphics animationprocessor 108 and image rendering engine 110 can include any suitablehardware, and each can be optimized for graphics-intensive computingwith the incorporation of one or more graphics processing units (GPUs).Communication interfaces for the processors 102, 106, 108 and renderingengine 110 can be any suitable network communication interface, but canbe optimized in some embodiments for high speed data transfer betweenthe graphics processing devices 106, 108 and 110.

Referring now to FIGS. 6-10B, methods are illustrated methods ofmodifying the images of respective original structures to produce anelectronic representation of the one or more structures modified to showthe respective deemed damage. FIG. 6 shows an exemplary image 600 ofactual roofing 602 of a structure, such as might be included in an image300 received from a user of the system 100. The image recognitionprocessor 106 of the system 100 can identify the image 600 as showing“roofing,” therefore the graphics animation processor can select imagetextures corresponding to “damaged roofing.” FIG. 7 shows an electronicrepresentation 700 of an exemplary damage texture 702 representing“damaged roofing,” which texture can be an actual photo of damagedroofing or a digital representation thereof. FIG. 8A shows an exemplaryfirst texture map 800 with a randomized distribution of damage areas 802corresponding in density to a first amount of deemed damage. In someembodiments, the shapes of the damage areas 802 included in a damagetexture may be arbitrary shapes or randomly generated shapes. In someembodiments, the shapes of the damage areas 802 included in a damagetexture can be predetermined based on representative types of damage.FIG. 8B shows the first damage texture 702 applied to the first texturemap 800 such that the damage texture appears in the randomized damageareas 802′ to produce an electronic representation 810 of a first levelof deemed damage. FIG. 9A shows an exemplary second texture map 900 witha randomized distribution of damage areas 902 corresponding in density(in this case, a higher density) to a second amount (in this case, ahigher amount) of deemed damage. FIG. 9B shows the first damage texture702 applied to the second texture map 900 such that the damage textureappears in the randomized damage areas 802″ to produce an electronicrepresentation 910 of the second, higher, level of deemed damage with agreater percentage of the texture representing damage texture 702.

FIG. 10A shows the original image 600 of the structure roofing of FIG. 6modified using the electronic representation 810 of the first level ofdeemed damage to produce an electronic representation 1000 of thestructure damage 1002 in accordance with the first level of deemeddamage. FIG. 10B shows the original image 600 of the structure roofingof FIG. 6 modified using the electronic representation 910 of the secondfirst level of deemed damage to produce an electronic representation1005 of high-level structure damage 1007 in accordance with the secondlevel of deemed damage.

Referring now to FIGS. 11-14 , additional methods are shown formodifying the images of respective original structures to produce anelectronic representation the one or more structures modified withrespective electronic representations of the respective deemed damage.FIG. 11 shows an image 1100 of an actual structure 1102 on a property,in this case a house. The image recognition processor 106 of the system100 can identify the image 1100 as showing a “house.” In someembodiments, one of the image recognition processor 106 and the graphicsanimation processor 108 can recognize the substructures of a typicalhouse or actual structure 1102 to identify walls 1104, windows 1106and/or roof 1108.

Referring now to FIG. 12 , after processing of the original image 1100by the image recognition processor 106, and optionally, theidentification of substructures 1104, 1106 and 1108, the system 100 canproduce a wireframe representation 1200 of the actual structure 1102. Insome embodiments the wireframe representation 1200 is produced by thegraphics animation processor 108 and in other embodiments it is producedby other parts of the system 100. The wireframe representation 1200 caninclude coordinate locations 1202 corresponding to portions of theactual structure 1102. In the example of FIG. 12 , the coordinatelocations 1202 correspond to vertices of the roof 1108 in the originalimage 1100.

As previously described, the system 100 can determine a respectivedeemed damage corresponding to each respective structure based on thedeemed seismic action. FIG. 13 shows an electronic representation 1300of a damaged wireframe produced by modifying the original wireframerepresentation 1200 of FIG. 12 in accordance with an assigned level ofdeemed damage received from the system processor 102. In someembodiments, the damaged wireframe 1300 is produced by displacingcoordinate locations 1202 from their original location 1302 (shown inbroken line) in wireframe 1200 to shifted locations 1304 (shown insolid), wherein the displacement between the original location 1302 andshifted locations 1304 is correlated to the assigned level of deemeddamage. In the example of FIG. 13 , only the roof structure is shownmodified; in other embodiments additional coordinate locations, e.g.,for walls, windows, etc., can also be displaced.

After modifying the wireframe representation 1200 of the original imageto produce the damaged wireframe representation 1300, the system 100 canproduce an electronic representation of the structure in accordance withthe deemed damage level by rendering texture on the damaged wireframerepresentation. In some embodiments, the texture rendering for theelectronic representation can be performed by the image rendering engine110. Further, in some embodiments, the electronic representation of thestructure can be further modified by replacing the original texture withan electronic representation of damage in the texture in accordance withthe assigned level of deemed damaged (e.g., as described in connectionwith FIGS. 6-10B. For example, FIG. 14 shows an electronicrepresentation structure 1400 of the structure 1102 of FIG. 11 damagedin accordance with the deemed damage level by rendering texture on thedamaged wireframe representation to represent areas of structural damage1402 and modifying the original texture with an electronicrepresentation of damaged texture 1404 in accordance with the assignedlevel of deemed damaged.

Referring now to FIGS. 15A-15C and FIGS. 16A-16E, there is illustrated asystem 1500 for display of an electronic representation of physicaleffects and property damage resulting from a parametric earthquake event(hereinafter “Earthquake Visualization” system or “EQV” system) inaccordance with another aspect. The EQV system 1500 can include hardwaresuch as that in system 100 of FIG. 1 , e.g., system server/processor102, image recognition processor 106, graphics animation processor 108,and image rendering engine 110. The EQV system 1500 can include someapparatus, software and processes performed at a user location and otherapparatus, software and processes performed at a remote location,wherein the user location and remote location are operably connected bya digital communication network, (e.g., communication network 104 ofFIG. 1 ). In the illustrated embodiment, the EQV apparatus, software andprocesses performed at the user location are shown in FIGS. 15A-15C, theEQV apparatus, software and processes performed at the remote locationare shown in FIGS. 16A-16E, and the interconnecting digitalcommunication network is a global communication network, for example theinternet. In other embodiments, the disclosed EQV apparatus, softwareand processes may be distributed differently between the user locationand the remote location. In still other embodiments, the disclosed EQVapparatus, software and processes may be distributed between the userlocation and several remote locations. In addition to the internet,alternative digital communication networks can be used with the EQVsystem 1500 as long as they enable digital communications between theuser location and the remote location.

Referring now specifically to FIG. 15A, a user initiates the EQV system1500 on a computing device at the user location. The computing devicecan be a mobile communication device including, but not limited to, asmart phone, tablet, smart watch, a computer including, but not limitedto, a desktop or laptop computer, or a home automation device including,but not limited to, an Amazon® Echo® or Google® Home® device. The userinitiates the EQV system 1500 by downloading application software ontothe computing device.

Summarizing some of the methods shown in FIGS. 15A-15C, with referenceto selected blocks of the diagram (as denoted): (Block 1502) Userinitiates EQV App on mobile device. Optionally, EQV App determinesuser's location using GPS. Optionally, EQV App gets user info. andcontact information.

User selects type of video desired in EQV App, including, but notlimited to: Inside of structure; outside of structure; yard; garage withcar(s); pool (Block 1506). Optionally, user takes video of property withmobile device camera using EQV App (Block 1508). Additionally, a usercan upload existing videos (Block 1510). Video is uploaded to EQVserver/processor 1514 using EQV App (Block 1512).

The process continues as shown in FIG. 15B (block 1516). The userselects SI/PGA using the EQV App (Block 1518). Optionally, userspecified actual value. Optionally, user selects “historical value” forSI/PGA equivalent to values for named historical earthquake event.SI/PGA uploaded to EQV server/processor 1514 using EQV App (Block 1520).

The user can select classifications or identify specific objects, suchas a type of car in a garage (Block 1522). The classifications oridentified specific objects can be uploaded to the EQV server/processor1514 (Block 1524). The user can select scene dimensions and othercharacteristics, such as property square feet and/or value of a property(Block 1526). The scene dimensions and other characteristics can beuploaded to the EQV server/processor 1514 (Block 1528). The user canselect additional scenario parameters (Block 1530). The additionalscenario parameters can be uploaded to the EQV server/processor 1514(Block 1532).

The process continues as shown in FIG. 15C (Block 1534). The user caninitiate scenario processing (Block 1536) at the EQV server/processor1538. EQV server/processor 1538 can be the same as or different from EQVserver/processor 1514. The user device can receive processed video(s)from the EQV server/processor 1538 (Block 1540). The user device canreceive real property damage prediction(s) from the EQV server/processor1538 (Block 1542). The user device can receive personal property damageprediction from the EQV server/processor 1538 (Block 1544).

The user device can receive a share link for one or more seismic damagevideo(s) (Block 1546). The user device can play the one or more seismicdamage video(s) and views damage prediction (Block 1548). The userdevice can download the one or more video(s) to save and share to othersusing the share link (Block 1550).

Summarizing some of the methods shown in FIGS. 16A-16E with reference toselected blocks of the diagram 1600 (as denoted): EQV server/processoris initiated (Block 1602) and EQV server/processor processes uploadedvideo according to designated SI/PGA (Block 1604). The user thendesignates the type of video uploaded (Blocks 1606, 1622, 1640, 1656).In other embodiments, the type of video uploaded can be determined bythe system, e.g., using image processing and/or machine learning.

Home interior video selected (Block 1606): Physics-based motion effectsapplied to video image of the interior structure during simulated event(Block 1608). Physics-based cracking effects applied to image of walls(Block 1610). Physics-based breakage effects applied to image of windows(Block 1612). Physics-based damage applied to image of ceilings (e.g.,full or partial collapse) (Block 1614). Physics-based movement effectsapplied to image of furnishings (Block 1616).

The diagram continues in FIG. 16B (Blocks 1618 and 1620). Home exteriorvideo selected (Block 1622): Physics-based motion effects applied toimage of the exterior structure during simulated event (Block 1624).Physics-based motion effects applied to images of trees and vegetationduring event (e.g., dropping leaves; falling limbs; falling trees)(Block 1626). Physics-based structural failure applied to image ofbricks, chimneys, fences (Block 1628). Physics-based cracking effectsapplied to image of walls (Block 1630). Physics-based breakage effectsapplied to image of windows (Block 1632). Physics-based damage effectsapplied to image of roof (e.g., full or partial collapse) (Block 1634).Physics-based earth movement effects applied according to designatedSI/PGA (e.g., landslide; house movement on slope, etc.) (Block 1636).

The diagram continues in FIG. 16C (Block 1638). Garage video can beselected (Block 1640) the same as the home interior, but addphysics-based damage effects to structure images during event duration(Block 1642). Physics-based cracking effects can be applied to wallimages (Block 1644). Physics-based breakage effects can be applied towindow images (Block 1646). Physics-based damage effects can be appliedto ceiling images (Block 1648). Physics-based movement effects can beapplied to furnishing images (Block 1650). Physics-based damage effectsapplied to image of car(s) from structural collapse of garage (1652).

The diagram continues in FIG. 16D (Block 1654). Pool video can beselected (Block 1656): Physics-based water slosh effect added to watersurface during simulated event (Block 1658). Physics-based crackingeffects applied to image of walls (Block 1660). Physic-based crackingeffects can be applied to decks (Block 1662).

Damage Predictors (optional): Real Property Damage Prediction can beperformed (Block 1664); EQC Server/Processor determines approximatesquare feet of house and/or value of house from data source(public/private) based on GPS location from EQV App. (Alternative) Userprovides estimated real property value via EQV App. EQV Server/Processorcalculated predicted value of real property damage for user-specifiedSI/PGA. Personal Property Damage Prediction can be performed (Block1666). A share link can be generated for the videos (Block 1668).

The diagram continues in FIG. 16E (Block 1670). As previously described,an image labeling or annotation tool may be incorporated into the systemto detect, identify, recognize, and/or mark objects in the video and/orstill images. The EQV Server/Processor counts personal property objectsrecognized in video (while adding effects), e.g., number lamps, numberof paintings, number of tables, number of chairs, cars in garage (ifapplicable), etc. (as many labels as desired). EQV Server/Processor setsvalue to each labeled object based on database of values. (Alternative)User provides estimated personal property value via EQV App. EQVServer/Processor calculates predicted value of personal property damagefor user-specified SI/PGA.

EQV server/processor downloads processed video to user device 1680 onEQV App (Block 1672). (Optional) EQV server/processor downloadspredicted value of real property damage (Block 1674) and/or personalproperty damage (Block 1676) to the user device 1680. The EQVserver/processor can download a share link (Block 1678) to the userdevice 1680.

After receiving the processed video from the EQV server/processor, theuser at user device 114 plays video or views still photos showingelectronic representation of structure motion and/or structure damage inaccordance with the parametric seismic event and the geographic locationof the property comprising the structures. Optionally, the user viewspredicted damage values. User selects new video or new SI/PGA. Repeatprocess if desired with different images, different locations, differentevent types and/or different event parameters.

Summarizing some of the methods relating to Effects Modeling inaccordance with additional aspects: Motion (shake) effects added toentire video image based on designated SI/PGA (i.e., speed and magnitudeare appropriate for PGA). Damage effects applied based on polygonmodeling, texture sampling and texture substitution. Object in video isselected. Object is digitized into original polygon model (shape) andsampled for original texture (color and pattern). Object's originalpolygon model is deformed to create damaged polygon model; amount ofdeformation is based on designated SI/PGA. Object's original texture isreplaced by damaged texture comprising original texture interspersedwith contrasting “feature texture” (e.g., can be specific to type ofdamage such as cracks, exposed lumber, missing shingles, etc.);proportion of feature texture mixed to original texture is based ondesignated SI/PGA. Damaged object image is formed from damaged polygonmodel covered with damaged texture. Original object image is thenoverlayed by damaged object image in video.

Referring now to FIG. 17 , in some embodiments a structure, such ashouse 302 or garage 304 shown in FIG. 3 , can include an interior roomsuch as that pictured in interior image 1700. The interior image 1700can be a still image or an image from a video. The interior image 1700can capture a scene that includes structural components including one ormore walls, such a first wall 1702, second wall 1704 including one ormore doors 1706, and third wall 1708 including one or more windows 1710,a floor 1712, a ceiling, or any other suitable structural component. Thescene of the interior image 1700 can also include personal items,including a television 1714 mounted on a wall (such as the first wall1702), a picture 1716 mounted on a wall (such as the first wall 1702), acoffee table 1718 at a center of the floor 1712, a couch 1720 on thefloor 1712 to a side of the coffee table 1718, a table 1722 against thethird wall 1708 under the window 1710, a candle 1724 on the coffee table1718, a vase 1726 on the table 1722, and any other suitable personalitems.

Referring now to FIG. 18A, there is shown an exemplary user interface1800. The first user device 114 can use the camera 116 to capture aninterior image 1700 of the scene shown in FIG. 17 . Alternatively, theinterior image 100 can be retrieved from storage. The interior image1700 is uploaded into the system either on a device or in a remotelocation. The user interface 1800 can be displayed on the display device120 of the first user device 114. The system can process the interiorimage 1700 to generate digital representations 1802-1826 for each of theobjects 1702-1726 shown in FIG. 17 for display on the user interface1800. The generation of the digital representations 1802-1826 can occurin the user device 114 or in a server 102 that the user device 114communicates with. The user interface 1800 can further include a damageselection window 1828 for selecting a type of damaging event tovisualize in the user interface 1800. The damage selection window 1828can include a flood input 1830, a seismic input 1832, a wind input 1834,or any other suitable input 1836. Once an input has been selected,additional windows related to the specific event can also be displayed.In certain embodiments, the additional windows are always displayed andthe values are changed based on the selection of the damage selectionwindow 1828.

Referring now to FIG. 18B, the flood input 1830 has been selected on thedamage selection window 1828 of the user interface 1800. Once the floodinput 1830 is selected from the damage selection window 1828, a floodtime window 1838 and water level selection window 1840 are displayed.The flood time selection window 1838 controls the time of the floodingevent displayed on the user interface. The flood time selection window1838 can have predetermined times set, such as a “during” input 1842 forshowing a representation of the room while the flooding event is inprogress, an “after” input 1844 for showing a representation of the roomafter the flooding event has ended but the water remains (shown selectedin FIG. 18B), a “drain” input 1846 for showing a representation of theroom after the water has been drained (shown in FIG. 18C), and timeselection input 1848 for entering an intermediate time during theflooding event. The water level selection window 1840 can providedifferent predetermined levels of flood intensity including a firstlevel input 1850 (shown selected in FIGS. 18B and 18C), a second levelinput 1852, a third level input 1854, “enter value” level input 1856, orany other suitable intensity level inputs. The predetermined levels canbe based on height of water for historic storms, be a percentage of theroom, or any other suitable way to determine a water level.

As shown in FIG. 18B, the flood input 1830 is selected on the damageselection window 1828, the after input 1844 is selected on the floodtime selection window 1838, and the first level input 1850 is selectedon the water level selection window 1840. The represented water level1858 corresponds to the first level input 1850. The represented waterlevel 1858 rises up the digital representation walls 1802, 1804, 1808,covers digital representation coffee table 1818, and partially coversthe digital representation couch 1820 and the digital representationtable 1822. As shown in FIG. 18C, the selection in the flood timeselection window 1838 is changed from the after input 1844 to thedrained input 1846. The water has been drained in the user interface1800. The represented digital representation walls 1802, 1804, and 1808now show digital representation mold 1860 or other damage. The digitalrepresentation couch 1820 can be made of a fabric and show digitalrepresentation stain(s) 1861. The digital representation table 1822could be made of wood and show digital representation rot 1862.

Referring now to FIG. 18D, the seismic input 1832 has been selected onthe damage selection window 1828 of the user interface 1800. Once theseismic input 1832 is selected from the damage selection window 1828, aseismic time window 1864 and seismic intensity window 1866 aredisplayed. The seismic time window 1864 controls the time of the seismicevent displayed on the user interface. The seismic time window 1864 canhave predetermined times set, such as during input 1842 showing the roomrepresentation with the seismic event is in progress, an after input1844 for showing the room representation after the seismic event hasended (shown selected in FIG. 18D), and time selection input 1848 forentering an intermediate time during the seismic event. The seismicintensity window 1866 can provide different predetermined levels ofrepresented seismic intensity including a first level, a second level1852, a third level 1854 (shown selected in FIG. 18D), an enter valuelevel 1856, or any other suitable intensity levels. The predeterminedlevels can be based on strength of an earthquake for historic quakeevents or any other suitable way to determine a seismic level or massivevibration level.

The user interface 1800, shown in FIG. 18D, displays a representation ofseismic damage that would be cause by an earthquake or other massivevibration event. The seismic damage caused a digital representationcrack 1868 in the digital representation first wall 1802 from theceiling or top of the digital representation first wall 1802. Thedigital representation picture 1816 is crooked, the digitalrepresentation television 1814 has fallen off the digital representationfirst wall 1802 and includes a digital representation crack 1870 in thescreen. The vibrations caused the digital representation vase 1826 tofall and break into digital representation pieces 1872. The digitalrepresentation candle 1824 fell off the digital representation coffeetable 1818. The digital representation couch 1820 has shifted.

Referring now to FIG. 18E, the wind input 1834 has been selected on thedamage selection window 1882 of the user interface 1800. Once the windinput 1834 is selected from the damage selection window 1828, a windtime window 1874 and wind speed selection window 1876 are displayed. Thewind time window 1874 controls the time of the represented wind eventdisplayed on the user interface. The wind time window 1874 can havepredetermined times set, such as a during input 1842 the wind event isin progress, an after input 1844 for after the wind event has ended(shown selected in FIG. 18E), and time selection input 1848 for enteringan intermediate time during the wind event. The wind speed selectionwindow 1876 can provide different predetermined levels of representedwind intensity including a first level input 1850, a second level input1852, a third level input 1854, enter value level input 1856 (shown inFIG. 18E), or any other suitable intensity levels. The predeterminedlevels can be based on strength of wind or category of historic stormsor any other suitable way to determine a wind strength level.

As shown in FIG. 18E, the after input 1844 is selected such that theuser interface 1800 displays the represented aftereffects of a winddamaging event. A specific value input 1856 has been entered in the windspeed selection window 1876. The value selected by the user was strongenough to shatter 1877 the digital representation window 1810 on thedigital representation third wall 1808. The represented wind caused thedigital representation picture 1816 to rotate and the digitalrepresentation couch 1820 to move back to the second wall 1804 blockingthe digital representation door 1806. The represented wind strengthknocked the digital representation candle 1824 off of the digitalrepresentation coffee table 1818. The represented wind also knocked thedigital representation vase 1826 off of the digital representation table1822 and break into digital representation pieces 1872.

As shown in FIGS. 18A-18E, the user interface 1800 can also include adestructive event control bar 1878. The destructive event control bar1878 controls a timeline of the destructive event in the user interface1800. The destructive event control bar 1878 can include a destructiveevent time indicator 1880 with a destructive event progress indicator1882, a destructive event play button 1884, and a destructive eventpause button 1886. The destructive event time indicator 1880 shows atotal amount of time that the destructive event occurs. This couldinclude amount of time for waters to recede and mold to form on thedigital representation walls 1802, 1804, 1808 during a flooding event.The destructive event progress indicator 1882 indicates the current timein the of the destructive event displayed on the user interface 1800.The destructive event progress indicator 1882 can be selectivelyadjusted to adjust the video of the destructive event on the userinterface 1800. The destructive event play button 1884 can start thevideo at the current time of the destructive event. The destructiveevent pause button 1886 can pause the video at the time of thedestructive event.

FIG. 19 is a block diagram of a system for display of an electronicrepresentation of physical effects and property damage resulting from aparametric earthquake event in accordance with various embodiments ofthe disclosure. As shown in FIG. 19 , there is illustrated a system 1900for display of an electronic representation of physical effects andproperty damage resulting from a parametric earthquake event inaccordance with one aspect. Unless otherwise noted, in this applicationthe terms “parametric earthquake event” and “parametric seismic event”have the same meaning. A parametric earthquake event is a set ofparameters and values for defining an earthquake event in terms ofscientific principles, e.g., physics and geological principles. Suchparameters can include, but are not limited to, date, time, epicenterlocation, hypocenter location (i.e., focus), duration, magnitude, peakground acceleration (“PGA”), and maximum shaking intensity (“SI”),maximum ground amplitude, mean ground amplitude, shaking frequency, soiltype, rock type and fault classification. A parametric earthquake eventcan include some or all parameter values measured during an actual,i.e., “historical” earthquake event, but can also include some or allparameters values selected by a user. The system 1900 includes a systemserver 1902 or computing device. The system server 1902 contains one ormore processors, RAM memory, storage units and communication interfacesfor sending and receiving data to and from other devices. The systemserver 1902 may comprise a single machine or multiple machines,including virtual machines resident on “cloud servers.” The systemserver 1902 is operably connected to a communication network 1904 and toone or more subsystems, e.g., an image recognition processor 1906, agraphics animation processor 1908 and/or an image rendering engine 1910.In some embodiments, the communication network 1904 is the internet;however, the communication network 1904 can be any type of network thatallows the system server 1902 and subsystems 1906, 1908 and 1910 tocommunicate with one another and with a user device 1914. In theembodiment illustrated in FIG. 19 , the server 1902 is directlyconnected to the subsystems 1906, 1908 and 1910; however, in otherembodiments some or all of the subsystems may be connected to the servervia the communication network 1904. In some embodiments, the subsystems1906, 1908 and 1910 can communicate directly with one another forimproved data transfer.

The image recognition processor 1906 can include or incorporate an imagelabeling or annotation tool (i.e., “labeling tool”). The labeling toolcan be used to process and label the images for bounding box objectdetection and segmentation so that the image is readable by machines. Insome embodiments, the labeling tool can utilize human assistance and inother embodiments the labeling tool can operate solely with machinelearning or artificial intelligence processes. In some embodiments,different image labeling tools may be used for processing images ofdifferent image types (e.g., interior images, exterior images, etc.).Using the image labeling tools, the various objects in the providedimage can be labeled for specific purposes. In some embodiments, labeledobjects can be selected for replacement by a computer graphic object(e.g., 2D sprite or 3D polygon), which can be moved on-screen andotherwise manipulated as a single entity, e.g., for purposes of eventanimation. In some embodiments, labeled objects may be classified intodifferent types or categories of objects for different purposes. Forexample, in some embodiments, labeled objects can be categorized forproperties relating to event animation, e.g., movable-type objects,bendable-type objects, breakable-type objects, waterproof-type objects,water damageable-type objects, etc. In other embodiments, labeledobjects can be categorized for properties relating to inventory ordamage assessment, e.g., table-type objects, chair-type objects,window-type objects, door-type objects, hanging art-type objects,TV/computer screen-type objects, etc. Such classification may beperformed by the image recognition processor 1906 or by anotherprocessor, e.g., the system processor 1902 or graphics animationprocessor 1908. The image labeling tools can use known orfuture-developed detection techniques for detection of the objectincluding, but not limited to, semantic, bounding box, key-point andcuboid techniques.

A seismic event database 1912 is operably connected to the server 1902,either via the communication network 1904 or directly. The seismic eventdatabase 1912 stores parametric seismic event data corresponding to oneor more earthquakes or other seismic events. The parametric seismicevent data can include, but is not limited to, values for the followingparameters: event name, event date, event epicenter location, eventfocus (i.e., hypocenter) location, event duration, event magnitude,event PGA, event maximum shaking intensity (“SI”), event maximum groundamplitude, event mean ground amplitude, and event shaking frequency foreach seismic event. The seismic events data in the seismic eventdatabase 1912 may be actual historical earthquake data, “relocated”earthquake data (i.e., where the majority of the data corresponds to ahistorical earthquake, but the epicenter/hypocenter location is changedto a different location) or hypothetical earthquake data specified by auser or otherwise generated. In some embodiments, the seismic eventdatabase 1912 can be located within the server 1902 or one of thesubsystems. In some embodiments, the seismic event database 1912 caninclude data and/or values from public or private earthquake reportingagencies, such as the U.S. Geological Survey.

User devices 1914, 1915 can connect to the system 1900 through thecommunication network 1904. The user devices 1914, 1915 can be mobiledevices such as mobile phones, tablets, laptop computers or they can bestationary devices such as desktop computers or smart appliancesincluding, but not limited to, smart televisions. In some embodiments,aspects of the current system 1900 may include downloadable software or“apps” resident on the user devices 1914, 1915 and/or non-downloadablesoftware (e.g., “cloud based software”) that remains resident on theserver 1902 or other elements of the system 1900 and is accessed by theuser device 1914 via a web browser or other network interface.

Using the user devices 1914, 1915, system users can upload images ofactual property via the network 1904 to the server 1902. The images ofactual property can be captured using a camera 1916 on the first userdevice 1914 or from images stored in memory 1918, e.g., a computermemory, hard drive, flash drive or other data storage technology. Theimages can be video programs, audiovisual programs and/or single ormultiple still photo images. User devices 1914 can also transmitadditional information to the server 1902 regarding the images,including, but not limited to, the geographic location of the propertyin the images (either entered by the user as text, captured via GPS orwireless location information on the user device 1914, or captured visgeotagging information on the image file), the address of the propertyin the images, the name and/or other contact information of the user, adesired parametric seismic event and/or desired seismic parameters to beused in creating a parametric seismic event. Desired seismic parameterscan include strength of the seismic event, duration of the seismicevent, etc. Each of the seismic parameters can be input on the userdevices 1914, 1915 by any suitable means. For example, the seismicparameters can be input or selected using a predetermined list, a sliderassociated with different values of the seismic parameters, a knob, anumber entry. The inputs can be physical components on the user device1914, 1915 or virtual representations on a display 1920 of the userdevice 1914, 1915.

The system 1900 processes, using an image recognition processor 1906,the received images and identifies one or more structures comprising theproperty. The system 1900 retrieves parametric data regarding adesignated seismic event from the seismic event database 1912. Thesystem 1900 defines a key data pair corresponding to the specifiedgeographic location and the designated seismic event. The system 1900determines key attributes relating the key data pair. The key attributescorrelate to how the parametric seismic data changes for distantgeographic locations (i.e., at a distance from the event). In someembodiments, the values of key attributes are determined basedprinciples known in physics and/or geology including, but not limited toprinciples of seismic attenuation, resonant vibration and/or soilbehavior factors. In some embodiments, the values of key attributes aredetermined based on seismic attenuation factors and the distance betweenthe property and the event center. In some embodiments, the values ofkey attributes are determined based on resonant vibration factors, theshaking frequency of the event, and the mass and/or resonant frequencyof structures. In some embodiments, the values of key attributes aredetermined based on soil behavior factors and the soil at the propertylocation, the event center, and/or at intervening geological features.Some key attributes may vary in direct (i.e., linear) proportion to thedistance between the geographic location of the property/structure andthe seismic event center, whereas other key attributes may varyaccording to reciprocal square of distance, logarithmic decay ofdistance or other mathematical functions relating to the distance. Insome embodiments, the values of key attributes are selected by a user.Other key attributes may vary depending on other factors such asintervening geological features rather than distance. The system 1900determines, using a computer such as the system processor 1902, valuesof a deemed seismic action at the specified geographic location bymodifying the parametric data for the designated seismic event using allrelevant key attributes. In some embodiments, multiple sets of values ofthe key attributes are predetermined. In this case, the system processor1902 prepares a plurality of videos each combination of key attributesfor each value of the key attributes.

In certain embodiments, the images of actual property can be capturedusing a camera 1916 on the first user device 1914 or from images storedin memory 1918, e.g., a computer memory, hard drive, flash drive orother data storage technology. User devices 1914 can also storeadditional information regarding the images, including, but not limitedto, the geographic location of the property in the images (eitherentered by the user as text, captured via GPS or wireless locationinformation on the user device 1914, or captured vis geotagginginformation on the image file), the address of the property in theimages, the name and/or other contact information of the user, a desiredparametric seismic event and/or desired seismic parameters to be used increating a parametric seismic event. Desired seismic parameters caninclude strength of the seismic event, duration of the seismic event,etc. Each of the seismic parameters can be input on the user devices1914, 1915 by any suitable means. For example, the seismic parameterscan be input or selected using a predetermined list, a slider associatedwith different values of the seismic parameters, a knob, a number entry.The inputs can be physical components on the user device 1914, 1915 orvirtual representations on a display 1920 of the user device 1914, 1915.

The processor 1922 on the user devices 1914, 1915 can process thereceived images and identifies one or more structures comprising theproperty. The processor 1922 can retrieve parametric data regarding adesignated seismic event from the seismic event database 1912 via thecommunication network 1904. The processor 1922 can define a key datapair corresponding to the specified geographic location and thedesignated seismic event. The processor 1922 can determine keyattributes relating the key data pair. The key attributes correlate tohow the parametric seismic data changes for distant geographic locations(i.e., at a distance from the event). In some embodiments, the values ofkey attributes are determined based principles known in physics and/orgeology including, but not limited to principles of seismic attenuation,resonant vibration and/or soil behavior factors. In some embodiments,the values of key attributes are determined based on seismic attenuationfactors and the distance between the property and the event center. Insome embodiments, the values of key attributes are determined based onresonant vibration factors, the shaking frequency of the event, and themass and/or resonant frequency of structures. In some embodiments, thevalues of key attributes are determined based on soil behavior factorsand the soil at the property location, the event center, and/or atintervening geological features. Some key attributes may vary in direct(i.e., linear) proportion to the distance between the geographiclocation of the property/structure and the seismic event center, whereasother key attributes may vary according to reciprocal square ofdistance, logarithmic decay of distance or other mathematical functionsrelating to the distance. In some embodiments, the values of keyattributes are selected by a user. Other key attributes may varydepending on other factors such as intervening geological featuresrather than distance. The processor 1922 can determine values of adeemed seismic action at the specified geographic location by modifyingthe parametric data for the designated seismic event using all relevantkey attributes. In some embodiments, multiple sets of values of the keyattributes are predetermined. In this case, the processor 1922 canprepare a plurality of videos each combination of key attributes foreach value of the key attributes.

The user device 1914, 1915 can also include a storage device 1924. Amemory and a persistent storage are examples of storage devices 1924,which represent any structure(s) capable of storing and facilitatingretrieval of information (such as data, program code, and/or othersuitable information on a temporary or permanent basis). The storagedevice 1924 may represent a random access memory or any other suitablevolatile or non-volatile storage device(s). The storage device 1924 maycontain one or more components or devices supporting longer-term storageof data, such as a read only memory, hard drive, Flash memory, oroptical disc.

The storage device 1924 can also include an AR plane manager application1926, an AR mesh manager application 1928, and an AR camera managerapplication 1930. The processor 1922 can run the AR plane managerapplication 1926, the AR mesh manager application 1928, and the ARcamera manager application 1930 for display or an electronicrepresentation of physical effects and property damage resulting frame aparametric natural disaster event.

The AR plane manager application 1926, run using the processor 1922, canextract AR planes from a scene. The AR planes can be 3D planes that canbe defined by a position, an orientation, and bounds. The AR planes canbe rendered on a display 1920. The AR planes can also be classified intocategories, such as wall, floor, ceiling, door, window, seat, table,none, etc.

The AR mesh manager application 1928 can mesh the Lidar point cloud inan AR wrapper. For example, the AR wrapper can be AR Foundation providedby Unity on platforms such as Apple iOS, such as iOS system ARKIT forVisual Inertial Odometry and 3D meshing, and Google Android. The 3D meshconsists of vertices, triangle definitions, and surface normal, as shownin FIG. 22 . The 3D mesh can be organized in a form of regions. Eachregion can contain a 3D mesh, where the 3D mesh can be managed andupdated independently from other 3D meshes.

The AR camera manager application 1930 can provide a tracked six degreeof freedom pose of a camera 1916 at every frame. The AR camera managerapplication 1930 can extract a camera image and camera parameters ateach frame for computer vision applications.

Although FIG. 19 illustrates an example of a system for display of anelectronic representation of physical effects and property damageresulting from a parametric earthquake, various changes may be made toFIG. 19 . For example, various components in FIG. 19 may be combined,further subdivided, replicated, omitted, or rearranged and additionalcomponents may be added according to particular needs.

Specifically, in some embodiments, the system for display of anelectronic representation of physical effects and property damageresulting from a parametric earthquake event can be provided in astand-alone mobile device such as, but not limited to, a mobile phone,tablet or similar consumer electronic device. In such embodiments, theuser device 1914 itself incudes the processor 1922, display device 1920,input device (e.g., touch screen), camera 1916 (including opticalcameras and, optionally, Lidar or other distance sensors), image storage1918 and storage device 1924. In such mobile device-based embodiments,the device 1914 may also include an image recognition processor 1906,graphics animation processor 1908 and/or image rendering engine 1910implemented in hardware or software on the mobile device instead of, orin addition to, processors accessed remotely through a network 1904 andserver 1902. In such mobile device-based embodiments, the device 1914may also include a seismic event database 1912 stored in the memorydevice instead of, or in addition to, a seismic event databaseaccessible via the network.

FIGS. 20A and 20B illustrate an example method for an AR earthquakeeffect visualization and assessment process 2000 in accordance with thisdisclosure. As shown in FIGS. 20A and 20B, the AR earthquake effectvisualization process 2000 is provided to produce a visualrepresentation of possible damage that would be caused in an earthquake.The earthquake effect visualization process can include a scenegeneration subprocess 2002 shown in FIG. 20A and a simulationsub-process 2004 shown in FIG. 20B.

As shown in the scene generation subprocess 2002 of FIG. 20A, theprocessor 1922 can scan a scene using an optical recognition method inoperation 2006. In certain embodiments, the scene can be an interior ofa room and one or more walls of the room can be scanned. The user device1914 can further include a time-of-flight camera 1916 or Lidar sensor,which the processor 1922 can operate to scan the one or more walls,ceilings and/or floors of the room. The processor 1922 can group anddetect scanned points from the scene into planes, which are stored inthe AR plane manager 1926. In some embodiments, the operation 2006includes detecting planes in the scene using the depth information fromthe optical camera and/or Lidar camera 1916. In some embodiments, thedetection of planes in the scene of operation 2006 further includesdetermining, using the information from scan, that planes detected inthe scene can be classified as one or more walls, ceilings and/orfloors. In some embodiments, the determination of planes detected in thescene further comprises using 3-D camera position information or 6-Dcamera position and orientation information in association with thedepth information.

The processor 1922 can also operate the camera 1916 to capture the wall,ceiling or floor as an orientation of the user device 1914 is rotate andmoved around the room. One or more sensors of the user device 1914 cancapture details for determining a camera pose for the camera 1916 aseach frame of the wall, ceiling or floor is captured. The camera posecan be saved along with the Lidar information or each could be storedseparately with a respective time stamp to match.

The processor 1922 can create a network mesh covering a scanned scene inoperation 2008. In some embodiments, the network mesh can be atriangular mesh. In some embodiments, the network mesh can be a pointcloud. In some embodiments, each surface or point of the mesh isdetermined based on a distance and/or orientation from an initialreference point. In some embodiments, each surface of the mesh isdetermined based on a distance and/or orientation from one or moreplanes in the scene. The network mesh can be determined based on acombination of the Lidar and the camera pose. In certain embodiments,the camera can be used in combination with the camera pose and othersensors on the user device 1914 to create the network mesh. The networkmesh can be formed using one or more shapes, such as a triangular mesh.

The processor 1922 can select a texture for an AR background inoperation 2010. The texture can be based on an image of the scenecaptured by the camera 1916. The processor 1916 can analyze the imageand determine a texture for the AR background. Different types oftextures that can be selected could include color, pattern, etc. The ARbackground can have one or more textures that can be selected. Incertain embodiments, the camera can process an image of a wall in a roomand determine a color and pattern for the wall to be used for thetexture of the AR background to simulate the color and pattern on theactual wall. Alternatively, the texture of the AR background can bepreselected on a user device, selected at the time of mesh generation,or read from a default setting in the memory. While a texture close tothe actual background of the scene is preferable in some embodiments, atexture that is different from the background can be chosen in someinstances to more clearly distinguish the AR background from objects inthe scene.

The processor 1922 can put the selected texture on the AR background inoperation 2012. Once the texture is selected based on operation 2010,the processor 1922 can build a mesh for a point cloud or a mesh for a 3Ddepth image as triangle definitions. The selected texture can be viewedon the display 1920 overlaid as texture on the generated mesh. This meshis virtually placed in the scene on top of the background currentlycaptured by the camera 1916 of the user device 1914. A dimension (e.g.,depth or distance from the camera) for the background captured from thecamera 1916 can be determined and the selected texture can beimplemented in front of or behind physical objects in the scene wherethe background may be not visible. Using the depth value associated witheach pixel in the background, the generated mesh with the selectedtexture can be placed relative (in front of/behind) to the objects inthe background. This can be used to selectively hide or show objects inthe scene.

The processor 1922 can replace an actual object in the scene with acorresponding AR shape (i.e., AR object) and at a corresponding locationin the scene in operation 2014. Objects in the scene can be identifiedin front of the background. Objects in the scene may include, but arenot limited to, all types of furnishings and personal property withinresidence or building such as tables, chairs, sofas, lamps, cabinets,chests, desks, books, bookcases, shelves, beds, stoves, ovens, microwaveovens, dishes, cups, glasses, window curtains, window blinds, pictures,artwork, clocks, vases, rugs, carpets, televisions, radios, electronicequipment (collectively “furnishings”). In some embodiments, the objectsin the scene can be identified by evaluating the previously-created meshfor specific predetermined patterns. Such predetermined patterns mayinclude, but are not limited to, mesh contours, mesh contour borders,mesh flatness, mesh depth from background (e.g., distance in front ofthe wall), mesh orientation from background (e.g., angle to wall). Suchpredetermined patterns may be stored in the memory 1924 of the userdevice 1914. In some embodiments, identified objects in the scene may beclassified into one or more categories of like objects. In someembodiments, the method may allow the user to confirm whether theclassification of the identified object in the scene is correct. In someembodiments, the user may be able to edit the classification of anidentified object. The processor 1922 can determine dimensions,orientation, and position within the scene for each detected object. Theprocessor 1922 can use the dimensions of the object to create acorresponding AR shape and use the orientation and position within thescene for placing the AR shape as an overlay for the object in thedisplay 1920. The AR shape can correspond to a 3D representation of theobject. The processor 1922 can extract and layout an actual image of theobject in operation 2016. The processor 1922 uses the camera 1916 tocapture one or more fames and can identify the object in the image frameusing the position information to identify the object scene. The portionof the image frame corresponding to the position of the object can beextracted. The extracted portion of the image can be overlaid on the 3Dshape as a virtual representation of the object. The virtualrepresentation of the object refers to the 3D shape with the actualimage overlaid. The actual image and the 3D shape can be manipulated asthe virtual representation of the object.

For example, in one embodiment the operation 2014 may evaluate the mesh,identify an object in a scene, and then classify the object into acategory of like objects, e.g., a framed artwork (i.e., “picture”), byevaluating one or more predetermined patterns in the mesh such as: a)rectangular shape, i.e., mesh face bounded by four straight contourlines; b) flatness, i.e., mesh face at a substantially constant (i.e.,within a preselected tolerance) distance above the underlying backgroundplane; c) shallowness, i.e., average distance of face mesh above theunderlying background plane is within a preselected tolerance; and/or d)frame presence, i.e., presence of a substantially constant (i.e., withina preselected tolerance) mesh color around the edges of the object. Insome embodiments, the method may request user confirmation that theobject is correctly classified as a framed artwork. Once the object isidentified as framed artwork, the processor 1922 can use the dimensionsof the actual object to create a corresponding AR shape with thecharacteristics of a framed artwork. The processor 1922 may use theorientation and position of the actual object within the scene forinitially placing the AR framed artwork as an overlay for the actualframed artwork in the display 1920. In other words, the AR object, inthis case a rectangle, may be initially displayed in front of the actualobject in the view shown on the display 1920. The AR background may bemodified to extend over the area coinciding with the actual object inthe view shown on the display 1920. A photo image of the actual objectmay be applied as a texture to the surface of the AR shape. Thus, the ARshape may initially appear to the user as the same shape and position ofthe actual object. The processor 1922 can subsequently move the positionof the AR framed artwork on the display screen 1920 and the image of theactual framed artwork will be obscured by the AR background so that itis not visible on the display screen. In this manner, the AR framedartwork will appear to the user observing the display screen 1920 to bethe actual framed artwork, but the AR framed artwork can be manipulatedby the processor 1922, e.g., to move, to oscillate, to fall and/or tobreak, despite the fact that the actual framed artwork remains unmovedand intact.

As shown in the simulation subprocess 2004 of FIG. 20B, the processor1922 can select one or more slider values in operation 2018. The slidervalues can be controls for a parameter or attribute of the seismicactivity. For example, the slider can control an intensity, a timeperiod, etc. Additional parameters or attributes of the seismic activitynot controlled by a slider can have values entered prior to thesimulation or read from a memory 1918. The processor 1922 can show aslider or other mechanism for selecting the parameter value or attributeof the seismic activity on the display 1920. In certain embodiments, theslider can control a current intensity and seismic activity wouldcontinuously occur at the selected current intensity until the currentintensity is set to zero on the slider. In some embodiments, atouch-screen or physical buttons, voice commands or other inputs can beused for entering parameters or attributes of the seismic activityinstead of, or in addition to, the slider.

The processor 1922 can generate simulated (e.g., AR) seismic activityfor the scene in operation 2020. The AR seismic activity can be shown tothe user through the display 1920 of the user device 1914. The ARseismic activity can cause the portion of the AR background that iscurrently visible to shift in the frame of view on the display 1920. Theedge of the background can be moved in and out of the frame in thedisplay according to the one or more slider values.

The processor 1922 can display simulated (i.e., AR) cracks in the ARbackground shown on the display 1920 based on the slider value inoperation 2022. In some embodiments, the processor 1922 can dynamicallygenerate AR cracks in the AR background during a simulated seismicevent, i.e., the cracks may grow in length or width as the seismic eventprogresses. In some embodiments, the processor 1922 can create a randomnumber of AR cracks with varying paths, widths and/or depths based onthe slider value in operation 2022. In some embodiments, the processor1922 can generate AR cracks featuring parallax shading to give thecracks the appearance of three-dimensional depth and width based on thepoint of view. The selected intensity of the simulation can be a factorfor the crack generation. The crack generation can also be determinedbased on an amount of time that has passed and/or a total time of thesimulation. For example, additional cracks may be formed as the time ofthe simulated seismic activity continues. In addition, the dimension ofthe cracks can change over time and based on the factors of the seismicactivity.

In operation 2024, the processor 1922 can move or manipulate an ARobject (i.e., the virtual representation of the object) shown on thedisplay 1920 during a simulation of a seismic event. In someembodiments, the processor 1922 may determine the type of a movement ora manipulation to be applied to an AR object based on one or moreseismic characteristics input by the user or received from the system.In some embodiments, the processor 1922 may determine the type of amovement or a manipulation to be applied to an AR object based on thecategory of like objects into which the object has been classified. Insome embodiments, the processor 1922 may determine the type of amovement or a manipulation to be applied to an AR object based onspecified or sensed characteristics of the AR object including, but notlimited to the object's mass, size and/or material. In some embodiments,the processor 1922 may determine the type of a movement or amanipulation to be applied to the AR object by applying a combination ofthe factors described above. Types of movements of the AR object thatmay be applied by the processor 1922 and shown on the display 1920include, but are not limited to, translation, rotation, oscillation,falling and rolling. Type of manipulations of the AR object that may beapplied by the processor 1922 and shown on the display 1920 include, butare not limited to, cracking, breaking, burning, leaking, sparking andsoiling. For example, if an object in the scene is classified as a typeof object that may be manipulated with “oscillation,” the processor 1922can show the AR object swinging back and forth about a hinge pointduring the seismic event, and the magnitude of the swinging may beselected based on a seismic characteristic set by the slider valueand/or on a characteristic of the object, such as its size. The virtualrepresentation of the object can swing in relation to the background ofthe scene. In addition, if there are multiple AR objects in a scene,each virtual representation may swing independently from other virtualrepresentations. In certain embodiments, a virtual representation of apicture frame can swing on a wall. While the virtual representation isswinging, the actual object is obscured by the wall set as a backgroundin order for the swinging motion to not be disrupted by the actualpicture frame.

In operation 2026, the processor 1922 can apply secondary manipulationtypes to the AR objects during a simulation of a seismic event. Asecondary manipulation can be any manipulation that is applied only whenpredetermined conditions are met during a simulated seismic event. Forexample, if the AR object has a primary manipulation type of“oscillation” during a seismic event, the AR object may have a secondarymanipulation type of “fall” if the magnitude of the shaking exceeds apreselected level, or if the shaking duration exceeds a preselectedperiod. Thus, the processor 1922 will initially show the AR objectoscillating (i.e., swinging) during a simulated seismic event, but willonly show the AR object falling if the preselected conditions are met.If the preselected conditions are not met, the AR object will continueswinging according to the primary manipulation type. In someembodiments, the processor 1922 may show the virtual representation ofthe object fall to the floor based on a highest slider value inoperation 2026. The virtual representation of the object canrealistically drop to the floor using standard physics for a fallingmotion of the virtual representation. When the virtual representation ofthe object reaches another object or a ground level, the virtualrepresentation can react with the other object or the ground level. Forinstance, a force could be applied to the other object based on thevirtual representation dropping to the floor. The virtual representationcan break or crack by the processor 1922 dividing the virtualrepresentation into two or more part virtual representations. Forinstance, a picture frame can reach the ground level and cause a crackthrough the picture frame creating two separate parts of the pictureframe. The two parts of the picture frame may operate independently fora remaining time of the simulation.

The virtual representation and background can be shaken independently.For example, a first picture frame can move independently from a secondpicture frame and a wall. The processor 1922 can create a set of parentcoordinate transforms in an application, such as Unity. The parenttransform can be an intermediate coordinate system between the worldcoordinate and children coordinates. The processor 1922 can set allsub-meshes from all regions that have a first face classification, suchas “floor”, to a first transform (also referred to as a parenttransform) and all sub-meshes from all regions that have a secondclassification, such as “wall”, to a second transform (also referred toas a child transform). The processor 1922 can set meshes that belong toa plane selected by a user as well as a plane with render cracks to athird transform. The processor 1922 can apply separate random shaking toeach of the parent transforms in order to simulate differentlyclassified virtual objects. Parent meshes ensure that a position of thesub meshes with respect to the parent transform remain constant.However, the position of the parent transform with respect to the worldis allowed to change. This allows all meshes assigned to a commonclassification to move together.

The shaking and falling of objects, such as picture frames, can beperformed by implementing Rigid body mechanics using a physic engine,such as but not limited to Unity Physics Engine, which is a standardgame engine functionality provided by Unity. The standard game enginecan simulate accelerations, connections, joints, and collisions betweenvirtual objects. The selected wall plane and each picture frame can beassigned rigid body functionalities, which allows the objects tointeract with each other in forms such as collisions, forces,constraints, and accelerations. The physics engine can provide functionsthat allow rigid bodies to relate to each other in a form ofconstraints. The physics engine can then iteratively solve multi-bodysimulation to give a position of the rigid bodies given the currentposition, constraints, and external forces/accelerations. The overallsteps for this procedure can include (1) assigning rigid bodyfunctionalities to various identified AR objects in the scene, (2)adding constraints, such as joints, links, etc. between the instantiatedvirtual objects, and (3) adding acceleration or movement to one or moreAR objects.

Although FIGS. 20A and 20B illustrate an example method for anearthquake effect visualization process 2000, various changes may bemade to FIGS. 20A and 20B. For example, while shown as a series ofsteps, various steps in FIGS. 20A and 20B may overlap, occur inparallel, occur in a different order, or occur any number of times.

FIGS. 21A and 21B illustrate an example method for scene generationprocess 2002 in accordance with this disclosure. As shown in FIG. 21A,the processor 1922 can activate AR plane manager application 1926 inoperation 2100. The AR plane manager application 1926 can create, updateand remove objects with AR plane components.

The processor 1922 can scan planes in a scene in operation 2102. Theplanes can be converted to AR planes by the AR plane manager application1926. The scanning of the planes can be performed by a Lidar sensor anda camera sensor. While the planes are scanned, other sensors on the userdevice 1914 can detect different information to be used with lidarinformation and camera information. Details of the AR plane can includeposition information, orientation information, bound information, etc.The planes can be classified by the AR plane manager application 1926.Non-limiting examples of classifications of the planes can include wall,floor, ceiling, door, window, none, etc.

The processor 1922 can select one or more planes to simulate anearthquake in operation 2104. The plane can be selected based on acurrent frame captured by the camera, selected by a user, etc. Incertain embodiments, the planes can be selected based on a number ofobjects associated with the plane, between the camera and the plane,etc.

The processor 1922 can activate an AR mesh manager application 1928 inoperation 2106. The AR mesh manager application 1928 can manage networkmeshes generated by the user device 1914. The AR mesh managerapplication 1928 can create, update, and/or remove virtual objects inresponse to the environment.

The processor 1922 can scan for meshes in operation 2108. The AR meshmanager application 1928 can be used to manage, store in memory, orperform operations on the meshes scanned by the processor. The meshescan be defined and/or categorized by the AR mesh manager application1928 as belonging to various object classes such as but not limited towall, floor, door, window, seat, none, etc. Meshes can be associatedwith objects in the scene in addition to a background. For example,meshes can be defined as part of a wall and part of one or more pictureframes on the wall.

The processor 1922 can activate an AR camera manager application 1930 inoperation 2110. The AR camera manager application 1930 can providetexture information and light estimation information. The AR cameramanager application 1930 can also provide camera calibration parameterssuch as focal length, principle point, etc. The AR camera managerapplication 1930 can extract a camera image and camera parameters (suchas camera pose parameters).

The processor 1922 can set global variables for the earthquake and alsodata extracted from the scanning process in operation 2114. The globalvariables can be received from a user selection on the display 1920 orinput of the user device 1914. Non-limiting examples of the globalvariables can include seismic intensity, seismic duration, direction ofseismic origin, etc. The global variables can also include variablesassociated with the selected plane, meshes that are not associated withthe plane, meshes that are associated with the plane, virtualrepresentations of object, etc.

The processor 1922 can extract a color of a selected plane in operation2116. In certain embodiments, the user device 914 can receive an inputfrom a user for selecting a color of the selected plane, such as a tapon the display 1920 of the user device 1914. The AR camera managerapplication 1930 can provide a camera image and the processor 1922 candetermine a location in the camera image corresponding to a location ofthe tap on the display 1920. The processor 1922 can determine a color atthe location in the camera image to extract. In certain embodiments, theprocessor 1922 can extract a color based on other factors, such as butnot limited to color at user tap position, an average color of the wall,a pattern on the wall, etc.

The processor 1922 can set a color for a mesh according to the extractedcolor in operation 2118. The AR camera manager application 1930 and/orprocessor 1922 can take the extracted color and apply the color to themesh corresponding to the background of the scene. In certainembodiments, the AR camera manager application 1930 can apply theextracted color to a wall in a room.

The processor 1922 can extract object suggestions from the mesh inoperation 2120. The processor 1922 can identify meshes corresponding toone or more objects in the scene. The meshes of different objects in thescene can overlap and the processor 1922 can distinguish differentobjects using the AR camera manager application 1930.

The processor 1922 can extract objects from the mesh belonging to aselected scene in operation 2122. Meshes that are not assigned to one ormore of the planes, backgrounds, walls, etc. can be determined by theprocessor to be objects. The objects can be individually processed bythe processor 1922 for details of the objects, such as dimensions,orientation, position, etc. The extraction of the objects from the meshis described in greater detail corresponding to the FIGS. 23A and 23B.

The processor 1922 can extract images for an object in a camera frame inoperation 2124. The processor 1922 can extract one or more camera imagessupplied by the AR camera manager application 1930. The camera imagescan be continuously captured based on movement of the AR camera in orderto capture the object from different angles. In certain embodiments, theprocessor 1922 can capture camera images including one or more pictureframes.

The processor 1922 can determine that vertexes of the objects areclearly visible in the camera frame in operation 2126. Using the ARcamera manager application 1930, the processor 1922 can identify eachvertex of an object shown in the camera image. Camera images where theobject is only partially visible can be skipped or used for clearlyidentified vertexes.

The processor 1922 can extract an image corresponding to the object inoperation 2128. The details of the objects can be used by the processor1922 to identify an actual object in the camera image and extract apartial image for the actual object. For example, the processor 1922 canidentify a picture frame in one or more camera images and make sure thatall the corners of the picture frame are clearly visible in each cameraimage. The extraction of the image corresponding to the objects isdescribed in greater detail corresponding to the FIGS. 24-26C.

The processor 1922 can instantiate selected objects in operation 2130.The processor 1922 uses the AR camera manager application to overlay theactual image of the object over the corresponding meshes associated withthe object in the AR scene. For example, an actual image of a pictureframe can be overlaid over meshes associated with the picture frame.

Although FIGS. 21A and 21B illustrate an example method for scenegeneration process 2002, various changes may be made to FIGS. 21A and21B. For example, while shown as a series of steps, various steps inFIGS. 21A and 21B may overlap, occur in parallel, occur in a differentorder, or occur any number of times.

FIG. 22 illustrates example meshes for assigning to a plane inaccordance with this disclosure. As shown in FIG. 22 , meshes can beassigned to a plane, such as in operation 2112. A goal of operation 2112is to separate meshes into belonging to a plane or not belonging to aplane, such as a wall. The AR mesh manager application 1928 can defineeach mesh, such as a first mesh 2200 and a second mesh 2202, as a listof vectors, such as a first vertex 2204, a second vertex 2206, a thirdvertex 2208, and a fourth vertex 2210, in a 3D space, a triangledefection, a face classification, etc. For example, the vectors in 3Dspace can be defined as ([0, 0, 0], [1, 0, 0,], [1, 1, 0], [0, 1, 0]).The triangle definition can be defined by the vertices as ([0, 2, 1],[0, 3, 2]), where the first mesh 2200 is formed of the first vertex2204, the third vertex 2208, and the second vertex 2206, and the secondmesh 2202 is formed of the first vertex 2204, the fourth vertex 2210,and the third vertex 2208. The face classifications can be defined aswall or none as shown in FIG. 22 .

The AR wall plane that was selected during the plane scan processprovides information such as plane position and orientation, planeboundaries in a plane space coordinate, and a 3D parametric planeequation [a, b, c, d] such that ax+by +cz+d=0 for all 3D points [x, y,z] that lie on the plane.

Using this information, the processor 1922 to perform operation 2112uses a function to separate mesh points from belonging to a room planeand not belonging to the room plane, depending on whether the pointssatisfy the following conditions. (1) Points can be at most 0.1 m from aplane surface. Therefore, the at most absolute value of distance of apoint [x1, y1, z1] to plane [a, b, c, d] is less than 0.1 m. (2) Thepoint, lies within the plane boundaries to check if a point lies insidethe plane boundaries. First transform the point from world space toplane space. Let x_W=[x1, x2, x3] be the point in world space andx_P=[xp1, xp2, xp3] be the point in plane space, R_p be the rotationmatrix of the plane in world space, and P=[P1, P2, P3] be the positionof the plane in world coordinates. Then the transformation shown inequation 1 holds and the inverse transformation is shown in equation 2.x_W=R_p*x_P+P  (1)x_P=Inverse(R_p)*(x_W−P)  (2)

(3) The processor 1922 can check whether the point in plane space x_Plies inside the boundaries of the plane. The y coordinate of the pointcan be dropped in order to obtain a corresponding 2D coordinate of thepoint in the plane space. i.e. x_P=[xp1, xp2, xp3] [xp1, xp3] in the 2Dplane space, since the normal of the plane is the local y axis. (4)Finally to find out whether the point x_P belongs to the plane oroutside the plane, the processor 1922 can compute the Winding Number forthe points. If the winding number of the point does not equal to 0, thenthe point lies inside the plane boundaries. If the winding number of thepoint equals 0, the point lies outside the plane boundaries.

Although FIG. 22 illustrates an example meshes for assigning to a plane,various changes may be made to FIG. 22 . For example, various componentsin FIG. 22 may be combined, further subdivided, replicated, omitted, orrearranged and additional components may be added according toparticular needs.

FIGS. 23A and 23B illustrate an example object extraction 2300 from meshin accordance with this disclosure. As shown in FIG. 23A, an object,such as a picture frame, can be extracted from a mesh. From operation2112, the meshes can be categorized as a wall mesh 2302 or a none mesh2304 at this point. Using the AR mesh manager 1928, the processor 1922can get the classification for each triangle in the mesh. The processor1922 can identify the triangles that are classified as “none”. Thebounds of the disjointed meshes 2306 that have a classification of“none” can be used as initial bound cuboid 2308 as picture frameproposals.

The processor 1922 can use a modified version of the union findalgorithm with path optimization to determine the bounds 2306 of thedisjointed meshes. The union find algorithm was originally designed tosolve Kruskal's minimum spanning tree algorithm. The main goal of theunion find algorithm is to have an efficient method of extracting andstoring disjoint sets of nodes in a tree. The processor 1922 can findthe disjoined meshes 2306 of vertices inside a mesh that belongs to theclass “none”. As a non-limiting example, two picture frames exist insideof a single Mesh definition and need to be separated into twosub-meshes. The processor 1922 can use union operation to group twonodes into a single group and a find operation to find a parent of thegroup to which a node belongs to. This method can be adapted byassigning nodes to each vertex 2204-2210 in the mesh as a node and usethe triangle definitions as union operations as shown in FIG. 23B.

As shown in FIG. 23B, each vertex 2204-2210 is assigned individualparent meshes. The goal is to set the parents of those vertices that arejoined by a line to a common parent. According to the triangledefinition [0, 3, 2] results in lines joining the indices [0, 3], [3,2], [0, 2], which leads to join operations [0, 3], [3, 2], [0, 2]. Thefirst step uses union operation [0, 2] to assign the third vertex 2208as a child of the first vertex 2204. The second step uses operation [3,2] to assign the second vertex 2206 as a child of the third vertex 2208.The third step uses operation [0, 3] but both have already been assignedto the group making this operation redundant. This operation is repeatedfor each triangle with a face classification as “none”. The maximumbounding cube 2308 can be computed by extracting the maximum span of allthe vertices of each sub-mesh. The bounding cube 2308 is then displayedto the user as a prospective object, such as a prospective pictureframe.

Although FIGS. 23A and 23B illustrates an example object extraction 2300from mesh, various changes may be made to FIGS. 23A and 23B. Forexample, various components in FIGS. 23A and 23B may be combined,further subdivided, replicated, omitted, or rearranged and additionalcomponents may be added according to particular needs.

FIG. 24 illustrates an example method for a crack creation algorithm2400 in accordance with this disclosure. As shown in FIG. 24 , a crackcreation algorithm can utilize shader processes that rely on modernmobile GPUs for efficient parallel processing, allowing real-timedisplay of many previously expensive graphics algorithms. The crackcreation algorithm used here is performed in two main passes. The firstpass utilizes Voronoi noise functions to create a crack-like grayscaledepth map. This pass solely requires standardized UV coordinates as aninput (as well as custom variables for modifying noise density, width,etc). This shader pass is saved to memory via a texture buffer (known inUnity as a custom render texture [CRT]) to be sent to the second shaderpass. The second pass utilizes a technique known as parallax occlusionmapping (POM) to cheaply create a sense of 3D depth. As inputs, thispass requires mesh UVs, mesh normal vectors, camera view direction, anda texture (the CRT from the initial pass), as well as depth amountcustom parameter. POM requires a depth texture for sampling and thus itis necessary to create a texture in the first pass that is then passedto the POM algorithm in a second pass.

The crack creation algorithm 2400 can utilize UV mapping 2402 to projectthe shader algorithm 2404 onto the chosen mesh. The UV mapping algorithm2402 is described in greater detail in FIGS. 25A and 25B. The Voronoinoise algorithm 2404 is described in greater detail in FIGS. 26A-26C.

The crack creation algorithm 2400 can utilize a mesh 2408 and a cameraimage 2410 to form pseudo 3D cracks on a plane. The crack creationalgorithm 2400 can obtain mesh information such as UV coordinates andnormal (x, y, z) vectors 2412 from the chosen mesh 2408. The cameraimage 2410 can include other details, such as camera pose information,that can be used to determine a view direction 2414 that is applied inthe parallax occlusion mapping (POM) pass.

Voronoi noise is a noise function that utilizes distance functionsbetween tiled points to obtain consistent cell like boundaries andspaces. To create a randomized domain of the Voronoi noise, the Voronoinoise pass algorithm 2405 can choose a random seed that connects tomultiple randomizers. For example, the random seed can be chosen basedon a time in the application. The randomizers can take a standardized UVinput (linear gradient from 0-1 on x and y axis) and offset/scale thestandard UV coordinates (0-1) randomly in order to generate differentvariations of noise for displacement. To obtain more vertical orhorizontal cracks, the scaling of the UV coordinates can be adjustedseparately. These UV values are plugged into two Voronoi noisegeneration subshaders, a subshader used for the U values 2504 (X-axis)and a subshader used for the V values 2506 (Y-axis). The two subshaderscan be centered around 0 (−0.5 to 0.5) and combined into a single imageconsisting of U displacement on a red channel and V displacement on agreen channel. The processor 1922 can use a multiply value to scale adisplacement and/or distortion amount to be applied to the main Voronoigeneration. The displacement/distortion amount can be added to aninitial UV coordinate of the mesh geometry in a process known as domainwarping. This warped UV space is then plugged into a final Voronoisubshader as the UV input. The processor 1922 can clamp the output usinga smoothstep function to create sharp cell-like edges. This is savedinto the memory via a custom render texture feature from Unity.

The custom render texture algorithm 2418 can allow an output of a shaderto be saved into a texture and passed to various locations with Unityand allow for multiple passes. Within the crack plane creation script2400, the Voronoi depth shader can be called with desired parameters torender into a texture via the custom render texture algorithm 2418.

The texture from the custom render texture algorithm 2418 can be used asan input to the parallax occlusion mapping (POM) pass algorithm 2406.The POM pass algorithm 2406 can apply shader to a mesh material 2416 asa final output to be shown with the display.

Although FIG. 24 illustrates an example method for scene generationprocess 2002, various changes may be made to FIG. 24 . For example,while shown as a series of steps, various steps in FIG. 24 may overlap,occur in parallel, occur in a different order, or occur any number oftimes.

FIGS. 25A and 25B illustrate an example UV mapping algorithm 2402 forthe crack creation algorithm 2400 in accordance with this disclosure. Asshown in FIGS. 25A and 25B, UV checker maps 2500 and 2501 are examplesfor showing how to fit a texture onto a rectangular surface whilekeeping an aspect ratio. In certain embodiments, the UV checker map canbe replaced by the crack shader. In certain embodiments, crack width anddepth can be adjusted based off a scaling amount of the UVs to cheaplyfit the shader onto various shaped walls. UV mapping can be used toproject a procedurally generated texture onto the selected wall planegeometry from a capture process. The captured geometry can designate UVcoordinates to a specific location on the mesh where 2D textures can bemapped. In certain embodiments, the values for the 2D texture can bebetween 0.0 and 1.0.

The processor 1922 can hold the UV blocks 2502 as a steady size.Therefore, a number of UV blocks 2502 is proportional to a size of thewall and, as the wall size increases, more UV blocks 2502 are utilized.UV blocks 2502 can be normalized between 0 and 1 in order to keep aconstant aspect ratio. To normalize, an absolute value of the minimumvalue for the U value 2504 and V value 2506 to the actual values of theU value 2504 and the V value 2506 for each UV block 2502. The actualvalues of the U value 2504 and the V value 2506 for each UV block 2502can be divided by a maximum value between the U values 2504 and the Vvalues 2506. For example, the maximum value for the U value 2504 is 3.0and for the V value 2506 is 2.0, which means the maximum value used todivide with is 3.0 resulting in a normalized UV table 2508 of normalizedUV blocks 2510.

Although FIGS. 25A and 25B illustrate an example UV mapping for thecrack creation algorithm, various changes may be made to FIGS. 25A and25B. For example, various components in FIGS. 25A and 25B may becombined, further subdivided, replicated, omitted, or rearranged andadditional components may be added according to particular needs.

FIGS. 26A-26C illustrate an example Voronoi noise pass algorithm 2405for the crack creation algorithm 2400 in accordance with thisdisclosure. While the Voronoi noise pass algorithm 2405 is describedbeing performed by processor 1922, the Voronoi noise pass algorithm 2405can also be performed as a shader pass using a parallel computationgraphic processing unit. In particular, FIG. 26A illustrates anindividual cell distance function 2600, FIG. 26B illustrates a neighborcell distance function 2602, and FIG. 26C illustrates a Voronoi block2604. As shown in FIGS. 26A-26C, a Voronoi noise pass algorithm 2405 canbe used to define and delineate proximal regions around UV blocks 2502using the boundaries of the UV blocks 2502. The processor 1922 cansubdivide the normalized UV block 2510 from the UV table 2508 to getsmaller square cells 2606. The processor 1922 can randomly generate apoint 2608 in the cells 2606. A distance can be calculated for eachpixel in the cell 2606 from the point 2608 in the respective cell 2606.

The Voronoi noise pass algorithm 2405 can generate a usable seamlessdistance function by searching neighboring cells 2610. The processor1922 can calculate a distance from each pixel to a nearest point eitherin a respective cell or a neighboring cell. This distance calculation isperformed to determine a minimum distance to a point from the currentpixel across all searched cells. The neighbor cell distance function2602 is shaded to reflect this distance calculation for each pixel tothe closest point in any cell, as shown in FIG. 26B.

As shown in FIG. 26C, the Voronoi block 2604 has the borders from thecells 2606 redrawn to show Voronoi cells 2612. Instead of a search ofone cell 2606 around a current cell, a more accurate two cell search canbe performed. When points in neighbor cells are close together, a onecell radius can lead to distortions near borders. To create an edgeeffect on the Voronoi cells 2612, a second closest point is trackedalong with the closest point to calculate the distance to cell borders.A distance function of the second closest point can be assigned avariable F2 and a distance function of the closest point can be assigneda variable F1. Using F1 and F2, the processor 1922 can determinepseudo-border on the Voronoi cells 2612 with help of a smoothstepfunction and/or clamping function.

Although FIGS. 26A-26C illustrate an example Voronoi noise 2600 for thecrack creation algorithm, various changes may be made to FIGS. 26A-26C.For example, various components in FIGS. 26A-26C may be combined,further subdivided, replicated, omitted, or rearranged and additionalcomponents may be added according to particular needs.

FIG. 27 illustrates an example AR earthquake visualization andassessment 2700 in accordance with this disclosure. In particular, FIG.27 illustrates the user's point of view as the user holds a mobiledevice directed towards a portion of the room (the user's hand is notshown). The system is shown generating an AR earthquake scene on thedisplay screen based on the actual room scene visible behind the mobiledevice. As shown in FIG. 27 , The AR earthquake visualization andassessment 2700 can include a scene 2702 and a user device 2704. Thescene 2702 includes a wall 2706, a floor 2708, a window 2710, a firstpicture 2712, a second picture 2714, a chair 2716, and a rug 2718.However, the scene 2702 can include any other type of object and anynumber of a specific object.

The user device 2704 can include a display 2720, one or more sensors2722 (e.g., optical camera, time of flight camera, etc.), and one ormore physical inputs 2724 (e.g., buttons, switches, etc.). The userdevice 2704 can use a scan area 2723 one or more sensors 2722 to scanthe scene 2702 and detect the background (e.g., wall 2706 and floor2708) and objects (a window 2710, a first picture 2712, a second picture2714, a chair 2716, and a rug 2718). The user device 2704 can createvirtual or AR representations of the scene 2702 and presentrepresentations on the display 2720. For example, the display 2720 showsan AR wall 2726, an AR floor 2728, an AR window 2730, an AR firstpicture 2732, an AR second picture 2734, an AR chair 2736, and an AR rug2738 corresponding to the respective backgrounds 2706 and 2708 andobjects 2710-2718.

The AR backgrounds and AR objects can have characteristics individuallydetermined applied during and/or after the scanning process. Forexample, a weight, material, roughness factor, etc. can be estimated foran AR object. The characteristics can be used for each manipulation ofthe AR background and the AR objects. In certain embodiments, a valuecan be assigned to each of the AR backgrounds and the AR objects. Thevalue can be based on a purchase value, an actual value, an input value,a replacement value, etc. The values can be stored in a table, such asinventory table 2800 shown below in FIG. 28 , in the memory of the userdevice 2704, which can be viewed separately or in combination with theAR environment. The values can also be shown with AR objects as each ARobject is selected or as the value is determined. The values that areshown with the AR objects can disappear after a period of time orapproval by the user. The values can be adjusted by the user.

Values for damage can be determined based which of the following AReffects are applied to a respective AR background or AR object. Thevalues for damage can be based on a repair value, a replacement value,etc. The values for damage can be calculated based on an estimatedamount of damage and for an amount of damage exceeding a threshold, anAR object can be considered fully damaged or destroyed. The values fordamage can be saved in the table in the memory of the user device 2704.The damage values can be shown on the respective AR background or ARobject that receives the damage. The damage values can disappear after aperiod of time or selection from a user. The damage values can include atotal damage for the seismic activity applied to AR representation ofthe scene 2702.

As shown in FIG. 27 , the AR background and AR objects can bemanipulated on the display 2720 while remaining unaffected in the scene2702. As the user device 2704 is moved around the room, the display 2704updates a current status of the objects. The one or more sensors 2722can capture the actual room and the display can show the room in acurrent state of seismic activity. The display 2720 can also displaycontrols (such as a seismic attribute control 2740 and a start button2742. The seismic attribute control 2740 can an input 2744 that can beadjusted. The adjustment can control the seismic attribute, such asintensity, duration, etc., by controlling the input 2744. In theillustrated example, the input 2744 can be a slide mechanism, valueentry, knob, etc. The input 2744 can be controlled on a touchabledisplay or using the one or more physical inputs 2724. The start button2742 initiates a seismic event on the display based on the input 2744provided on the seismic attribute control 2740.

The display 2720 can apply a vibration or shaking effect 2746 to an ARbackground or an AR object, such as the AR wall 2726 and the AR floor2728. The shaking effect 2746 can be applied equally to the ARbackgrounds or can be applied stronger or weaker to the specific ARbackgrounds. The shaking effect 2746 applied to the AR background can beadditionally applied to each of the AR objects in a manner of equalcharacteristics or reduced effectiveness characteristics. The reducedeffectiveness characteristics can include lower shaking intensitytransferred from the background, lower resistance (increased duration ofshaking), etc. An amount of the vibration for the AR background or ARobject can be controlled based on the input 2744 to the seismicattribute control 2740.

The display 2720 can apply a cracking effect 2748 on an AR background oran AR object, such as the AR wall 2726. The cracking effect 2748 cancause a 3D crack to form in the AR background. The 3D crack can beviewed from different angles to show a depth to the 3D crack. Thecracking effect 2748 can include a 3D crack that extends from a first ARbackground to a second AR background. For example, the cracking effect2748 can begin on the AR wall 2726 and extend to an AR ceiling, an ARsecond wall, and/or the AR floor 2728. A 3D crack on an AR background,such as the AR wall 2726, can extend behind and hidden an AR object tocontinue extending past the AR object in view on the display 2720. Asize and growth rate for the 3D crack on the AR background or AR objectcan be controlled based on the input 2744 to the seismic attributecontrol 2740.

The display 2720 can apply a swinging effect 2750 to an AR object, suchas the AR first picture 2732. The swinging effect 2750 can cause the ARobject to shake similarly to the AR background contacting the AR objectdue to a fixed pendulum that the swinging occurs about. The swingingeffect 2750 on the AR object can include a translation component equalto the vibration or shaking effect 2746 of the AR background. A swingrate for the swinging of AR object can be controlled based on the input2744 to the seismic attribute control 2740.

The display 2720 can apply a falling effect 2752 to an AR object, suchas the AR second picture 2734. The falling effect 2752 can move the ARobject separately from the AR background and other AR objects. Thefalling effect 2752 can be based on gravity for controlling a fallingrate and/or a falling speed. The falling effect 2752 can use anestimated or determined weight of the AR object to determine a forcethat could be applied to other AR objects and/or AR backgrounds. Atiming for the falling of AR object can be controlled based on the input2744 to the seismic attribute control 2740.

In certain embodiments, a secondary effect can be applied to an ARobject, such as the AR second picture 2734. For example, a fallingeffect on an AR object causes the AR object to forcefully contact an ARbackground, such as the AR floor 2728, or another AR object, such as theAR chair 2736. A secondary effect of the falling effect could be abreaking effect 2754. The display can apply the breaking effect 2754 tothe AR object, such as the AR second picture 2734. Other examples ofsecondary effects can include, a bouncing effect, a ricochet effect, amerge effect, striking effect, etc.

The breaking effect 2754 can cause the AR object to split into multipleAR partial objects. Each of the multiple AR partial objects can besmaller than the AR object before the split. In certain embodiments, themultiple AR partial objects can collectively be approximately equal tothe broken AR object. An amount of AR partial objects can be based on anoriginal size of an AR object, a force applied to the AR object duringsimulation from one or more other AR effects, etc.

A shatter effect 2756 can cause an AR object to have multiple AR partialobjects splits from a main AR object, such as AR window 2730 or an ARmirror. The shatter effect 2756 can be a primary effect from the seismicdamage or a secondary effect from another AR object forcefullyinteracting with the original AR object. For example, the AR window 2730can shatter due directly to the vibrations from a seismic event or froman AR object, such as the AR second painting 2734 or generation of arandom AR object, such as an AR branch of a tree, an AR power line,etc., falling through the AR object, such as AR window 2730.

Although FIG. 27 illustrates an example AR earthquake visualization andassessment 2700, various changes may be made to FIG. 27 . For example,various components in FIG. 27 may be combined, further subdivided,replicated, omitted, or rearranged and additional components may beadded according to particular needs.

FIG. 28 illustrate an example inventory tables for objects andbackgrounds identified in a scene 2702 in accordance with thisdisclosure. In particular, FIG. 28A illustrates an example objectinventory table 2800 and FIG. 28B illustrates an example backgroundinventory table 2801. As shown in FIG. 28A, an object inventory table2800 can be populated with AR objects identified in a scene scanningprocess above. The inventory table can include an object number 2802, anobject category 2804, an initial value 2806, a custom value 2808, anassigned value 2810, a damage factor 2812, and a damage value 2814. Incertain embodiments, a new table can be generated for each newlyidentified scenes. When a scene identifies a previously scanned scene,the original table corresponding to the previously scanned scene can beamended or added. For instance, the objects added to the previouslyscanned scene can be added to additional rows of the table. Objects thatno longer appear in the previously scanned scene can be removed from thetable, a note can be added to the object row in the table, or otherwisemarked as no longer in the previously scanned scene.

The object number 2802 can be an ordered number assigned for an object.The object number 2802 can be a unique number for the scanned scene. Incertain embodiments, objects in different scenes are given unique objectnumbers 2802 for any scene that the user device has captured. In certainembodiments, object numbers 2802 can be unique for a specific location.In certain embodiments, each location can have object numbers 2802assigned as series. For example, the object number 2802 can be fourdigits and the first two digits are assigned a grouping of scenes andthe second two digits are assigned in order for each scene.

The object category 2804 can be a general narrative for the identifiedobject. In order to determine an object category 2804, the user devicecan compare the captured image to a database of standard objects. Incertain embodiments, the identification of the object can be performedby a server sent an image, video, or point cloud of the identifiedobject. The object can be identified by any standard objectidentification method. The object category 2804 can be identified by theuser primarily or secondarily after one or more of the user device and aremote server cannot match a description to the object. The objectcategory 2804 can also provide a fragility characteristic of the object.For example, a vase on a table could provide a description of warningregarding possibility of destruction, damage, secondary damage, etc.

The initial value 2806 can be determined for each of the scannedobjects. The processor 1922 can determine an initial value 2806 based onthe object category 2804. The initial value 2806 can be a value of anobject determined by the user device or the remote server. The initialvalue 2806 could be a minimum value, a medium value, a mean value, amaximum value, etc. The initial value 2806 could be a wholesale value, apurchase value, a market value, a replacement value, etc.

The custom value 2808 can be input by a user. A prompt could be providedto the user after an AR object is selected for generation on thedisplay. The user could input the custom value 2808 using the physicalbuttons. The user could provide the custom value 2808 by scanning abarcode or other visual code for identifying an object. In certainembodiments, the custom value 2808 can be entered on another remote userdevice.

The assigned value 2810 is the value used for totaling a total value ofitems in a scene. The assigned value 2810 can be determined based onwhether a value is entered in the custom value 2808. For example, theassigned value 2810 can be the custom value 2808 when a value is enteredfor the input value 2808 or the custom value 2808 is greater than 0. Theassigned value 2810 can be the initial value 2806 when the custom value2808 is not entered or the custom value 2808 is approximately equal to 0or an insignificant value compared to the initial value 2806.

The damage factor 2812 can be a factor related to damage of the object.The damage factor 1812 can be determined based on a threshold levelearthquake, a historic level earthquake (for a region), a maximumearthquake, or an earthquake controlled on the display by the user. Thedamage factor 1812 can be a factor related to an object being consideredtotally destroyed or a factor of damage where an object is considereddestroyed. The damage factor 1812 could be a factor related to a chancethat an object is destroyed in an average or low damage producingearthquake. In certain embodiments, a scene 2702 can be scannedfollowing an actual earthquake and the damage factor 2812 can reflect anamount of actual damage of the identified objects. The damage factor canbe determined based on the seismic effects applied to the specifiedobject.

The damage value 2814 is a value for the amount of damage cause by theseismic event. The damage value 2814 can be determined based on theseismic effects applied to the specified object. The damage value 2814can be determined by multiple the assigned value 2810 by the damagefactor 2812.

As shown in FIG. 28B, a background inventory table 2801 can be populatedwith AR backgrounds identified in a scene scanning process above. Theinventory table can include a background number 2816, a backgroundcategory 2818, an area 2820 of the background, an initial unit value2822, a custom unit value 2824, an assigned value 2826, a damage factor2828, and a damage value 2830. In certain embodiments, a new table canbe generated for each newly identified scenes. When a scene identifies apreviously scanned scene, the original table corresponding to thepreviously scanned scene can be amended or added.

The background number 2816 can be an ordered number assigned for abackground. The background number 2816 can be a unique number for thescanned scene. In certain embodiments, background in different scenesare given unique background numbers 2816 for any scene that the userdevice has captured. In certain embodiments, background number 2816 canbe unique for a specific location. In certain embodiments, each locationcan have background number 2816 assigned as series. For example, thebackground number 2816 can be four digits and the first two digits areassigned a grouping of scenes and the second two digits are assigned inorder for each scene. The background numbers 2816 can also be uniquefrom the object numbers 2802.

The background category 2818 can be a general narrative for theidentified background. In order to determine a background category 2818,the user device can compare the captured image to a database of standardbackgrounds. In certain embodiments, the identification of thebackground can be performed by a server sent an image, video, or pointcloud of the identified background. The background can be identified byany standard background identification method. The background category2818 can be identified by the user primarily or secondarily after one ormore of the user device and a remote server cannot match a descriptionto the background.

The background area 2820 is an area of the background in the scene. Theprocessor 1922 can determine the background area 2820 during thescanning process. A time of flight camera can capture edges of abackground, such as a wall, and calculate a total size for thebackground area 2820. The background area 2818 can be any standardmeasurements units.

The initial unit value 2822 can be determined for each of the scannedbackgrounds. The processor 1922 can determine an initial unit value 2822based on the background category 2818. The initial unit value 2822 canbe a value per measurement unit of the background area 2820. The initialunit value 2822 can be a value of a background determined by the userdevice or the remote server. The initial unit value 2822 could be aminimum value, a medium value, a mean value, a maximum value, etc.

The custom unit value 2824 can be input by a user. A prompt could beprovided to the user after an AR background is selected for generationon the display. The user could input the custom unit value 2824 usingthe physical buttons. The user could provide the custom unit value 2824by scanning a barcode or other visual code for identifying a background.In certain embodiments, the custom unit value 2824 can be entered onanother remote user device.

The assigned value 2826 is the value used for totaling a total value ofitems in a scene. For the background assigned value 2826, the value is atotal of the initial unit value 2822 or custom unit value 2824multiplied by the background area 2820. The assigned value 2826 can bedetermined based on whether a value is entered in the custom unit value2824. For example, the assigned value 2826 can be the custom value 2824when a value is entered for the custom value 2824 or the custom value2824 is greater than 0. The assigned value 2810 can use the initial unitvalue 2822 when the custom unit value 2808 is not entered or the customunit value 2824 is approximately equal to 0 or an insignificant valuecompared to the initial unit value 2822.

The damage factor 2812 can be a factor related to damage of thebackground. The damage factor 1812 can be determined based on athreshold level earthquake, a historic level earthquake (for a region),a maximum earthquake, or an earthquake controlled on the display by theuser. The damage factor 1812 can be a factor related to an object beingconsidered totally destroyed or a factor of damage where an object isconsidered destroyed. The damage factor 1812 could be a factor relatedto a chance that an object is destroyed in an average or low damageproducing earthquake. In certain embodiments, a scene 2702 can bescanned following an actual earthquake and the damage factor 2812 canreflect an amount of actual damage of the identified backgrounds. Thedamage factor can be determined based on the seismic effects applied tothe specified background.

The damage value 2814 is a value for the amount of damage cause by theseismic event. The damage value 2814 can be determined based on theseismic effects applied to the specified background. The damage value2814 can be determined by multiple the assigned value 2810 by the damagefactor 2812.

Although FIGS. 28A and 28B illustrate example inventory tables forobjects and backgrounds identified in a scene 2702, various changes maybe made to FIGS. 28A and 28B. For example, various components in FIGS.28A and 28B may be combined, further subdivided, replicated, omitted, orrearranged and additional components may be added according toparticular needs.

FIG. 29 illustrates an example method for an AR earthquake visualizationand assessment process 2900 in accordance with this disclosure. As shownin FIG. 29 , the processor 1922 can scan a scene 2702 using an opticalrecognition method in operation 2902. In certain embodiments, the scene2702 can be an interior of a room and one or more walls 2706 of the roomcan be scanned. The user device 1914 can further include atime-of-flight camera 1916 or Lidar sensor, which the processor 1922 canoperate to scan the one or more walls 2706, ceilings and/or floors 2708of the room. The processor 1922 can determine a distance to differentpoints on the wall 2706, ceiling and/or floor 2708 using Lidar. In someembodiments, the operation 2902 includes detecting planes in the scene2702 using the depth information from the optical camera and/or Lidarcamera 1916. In some embodiments, the detecting planes in the scene 2702of operation 2902 further includes determining, using the depthinformation from scan, that planes detected in the scene 2702 are one ormore walls 2706, ceilings and/or floors 2708 present in the scene 2702.In some embodiments, the determining that planes detected in the scene2702 are one or more walls 2706, ceilings and/or floors 2708 present inthe scene 2702 further comprises using 3-D camera position informationor 6-D camera position and orientation information in association withthe depth information.

The processor 1922 can also operate the camera 1916 to capture the wall2706, ceiling or floor 2708 as an orientation of the user device 1914 isrotate and moved around the room. One or more sensors of the user device1914 can capture details for determining a camera pose for the camera1916 as each frame of the wall 2706, ceiling or floor 2708 is captured.The camera pose can be saved along with the Lidar information or eachcould be stored separately with a respective time stamp to match.

The processor 1922 can identify a background and objects in a scene 2702in operation 2904. The processor 1922 can scan the scene 2702 formeshes. The meshes can be defined and/or categorized by the processor1922. The meshes can also be associated with a specified plane orspecified planes by the processor 1922. Meshes can be associated withobjects in the scene in addition to a background. For example, meshescan be defined as part of a wall and part of one or more picture frameson the wall. In order to identify backgrounds, the processor 1922 canassign one or more meshes from a region to the plane. The processor 1922can use the extracted camera image to determine a plane for each of themeshes. The meshes can be classified and associated with a specifiedplane by the processor 1922.

The processor 1922 can extract object suggestions from the mesh. Theprocessor 1922 can identify meshes corresponding to one or more objectsin the scene 2702. The meshes of different objects in the scene canoverlap and the processor 1922 can distinguish different objects. Theprocessor 1922 can extract objects from the mesh belonging to a selectedscene. Meshes that are not assigned to one or more of the planes,backgrounds, walls, etc. can be determined by the processor to beobjects. The objects can be individually processed by the processor 1922for details of the objects, such as dimensions, orientation, position,etc.

The processor 1922 can count the objects in operation 2906. Counting theobjects can include taking an inventory of the objects in the scene 2702and placing the information in an inventory table 2800. The objects canbe assigned an object number 2802 that can be used to relate a row onthe table to an AR object by the processor 1922. A description 2804 ofthe objects can be determined while counting. The description 2804 ofthe objects can allow a user to quickly and easily determine which rowcorresponds to a specific AR object or object in a scene. Thedescription 2804 can include a name of the object, a position of theobject, characteristics of the object, etc. The counting of the objectscan also include an accounting of the objects and an estimated value2806 or other value can be determined.

The processor 1922 can create one or more AR backgrounds and one or moreAR objects from the background and objects identified in the scene 2702in operation 2908. The processor 1922 can extract a color of a selectedplane. In certain embodiments, the user device 914 can receive an inputfrom a user for selecting a color of the selected plane, such as a tapon the display 1920 of the user device 1914. The camera can provide acamera image and the processor 1922 can determine a location in thecamera image corresponding to a location of the tap on the display 1920.The processor 1922 can determine a color at the location in the cameraimage to extract. In certain embodiments, the processor 1922 can extracta color based on other factors, such as amount of the color in thescene, an average color of the wall, etc. The processor 1922 can set acolor for a mesh according to the extracted color. The processor 1922can take the extracted color and apply the color to the meshcorresponding to the background of the scene. In certain embodiments,the processor 1922 can apply the extracted color to a wall, a floor,and/or a ceiling in a room.

The processor 1922 can extract images for an object in a camera frame.The processor 1922 can extract one or more camera images supplied by theAR camera. The camera images can be continuously captured based onmovement of the AR camera in order to capture the object from differentangles. In certain embodiments, the processor 1922 can capture cameraimages including one or more picture frames. The processor 1922 candetermine that vertexes of the objects are clearly visible in the cameraframe in operation 2126. Using the AR camera, the processor 1922 canidentify each vertex of an object shown in the camera image. Cameraimages where the object is only partially in the camera image can beskipped or used for clearly identified vertexes. The processor 1922 canextract an image corresponding to the object in operation 2128. Thedetails of the objects can be used by the processor 1922 to identify anactual object in the camera image and extract a partial image for theactual object. For example, the processor 1922 can identify a pictureframe in one or more camera images and make sure that all the corners ofthe picture frame are clearly visible in each camera image. Theprocessor 1922 can instantiate selected objects in operation 2130. Theprocessor 1922 can overlay the actual image of the object over thecorresponding meshes associated with the object in the AR scene. Forexample, an actual image of a picture frame can be overlaid over meshesassociated with the picture frame.

The processor 1922 can receive seismic characteristics in operation2910. The seismic characteristics can be received from a user, read froma memory of the user device, received from another user device, receivedfrom a remote server. For example, seismic characteristics or seismicalgorithm can be stored in a memory of the user device. The user can bepresented an input, such as a slider, to control a seismiccharacteristic, such as a seismic intensity. The slider values can becontrols for a parameter or attribute of the seismic activity. Forexample, the slider can control an intensity, a time period, etc.Additional parameters or attributes of the seismic activity notcontrolled by a slider can have values entered prior to the simulationor read from a memory 1918. The processor 1922 can show a slider orother mechanism for selecting the parameter value or attribute of theseismic activity on the display 1920. In certain embodiments, the slidercan control a current intensity and seismic activity would continuouslyoccur at the selected current intensity until the current intensity isset to zero on the slider. In some embodiments, a touch-screen orphysical buttons, voice commands or other inputs can be used forentering parameters or attributes of the seismic activity instead of, orin addition to, the slider.

The processor 1922 can display seismic effects based on the seismiccharacteristics in operation 2912. The processor 1922 can generatesimulated (e.g., AR) seismic activity for the scene in operation 2020.The AR seismic activity can be shown to the user through the display1920 of the user device 1914. The AR seismic activity can cause theportion of the AR background that is currently visible to shift in theframe of view on the display 1920. The edge of the background can bemoved in and out of the frame in the display according to the one ormore slider values.

The processor 1922 can display simulated (i.e., AR) cracks in the ARbackground shown on the display 1920 based on the slider value inoperation 2022. In some embodiments, the processor 1922 can dynamicallygenerate AR cracks in the AR background during a simulated seismicevent, i.e., the cracks may grow in length or width as the seismic eventprogresses. In some embodiments, the processor 1922 can create a randomnumber of AR cracks with varying paths, widths and/or depths based onthe slider value in operation 2022. In some embodiments, the processor1922 can generate AR cracks featuring parallax shading to give thecracks the appearance of three-dimensional depth and width based on thepoint of view. The selected intensity of the of the simulation can be afactor for the crack generation. The crack generation can also bedetermined based on an amount of time that has passed and/or a totaltime of the simulation. For example, additional cracks may be formed asthe time of the simulated seismic activity continues. In addition, thedimension of the cracks can change over time and based on the factors ofthe seismic activity.

In operation 2024, the processor 1922 can move or manipulate an ARobject (i.e., the virtual representation of the object) shown on thedisplay 1920 during a simulation of a seismic event. In someembodiments, the processor 1922 may determine the type of a movement ora manipulation to be applied to an AR object based on one or moreseismic characteristics input by the user or received from the system.In some embodiments, the processor 1922 may determine the type of amovement or a manipulation to be applied to an AR object based on thecategory of like objects into which the object has been classified. Insome embodiments, the processor 1922 may determine the type of amovement or a manipulation to be applied to an AR object based onspecified or sensed characteristics of the AR object including, but notlimited to the object's mass, size and/or material. In some embodiments,the processor 1922 may determine the type of a movement or amanipulation to be applied to the AR object by applying a combination ofthe factors described above. Types of movements of the AR object thatmay be applied by the processor 1922 and shown on the display 1920include, but are not limited to, translation, rotation, oscillation,falling and rolling. Type of manipulations of the AR object that may beapplied by the processor 1922 and shown on the display 1920 include, butare not limited to, cracking, breaking, burning, leaking, sparking andsoiling. For example, if an object in the scene is classified as a typeof object that may be manipulated with “oscillation,” the processor 1922can show the AR object swinging back and forth about a hinge pointduring the seismic event, and the magnitude of the swinging may beselected based on a seismic characteristic set by the slider valueand/or on a characteristic of the object, such as its size. The virtualrepresentation of the object can swing in relation to the background ofthe scene. In addition, if there are multiple AR objects in a scene,each virtual representation may swing independently from other virtualrepresentations. In certain embodiments, a virtual representation of apicture frame can swing on a wall. While the virtual representation isswinging, the actual object is obscured by the wall set as a backgroundin order for the swinging motion to not be disrupted by the actualpicture frame.

In operation 2026, the processor 1922 can apply secondary manipulationtypes to the AR objects during a simulation of a seismic event. Asecondary manipulation can be any manipulation that is applied only whenpredetermined conditions are met during a simulated seismic event. Forexample, if the AR object has a primary manipulation type of“oscillation” during a seismic event, the AR object may have a secondarymanipulation type of “fall” if the magnitude of the shaking exceeds apreselected level, or if the shaking duration exceeds a preselectedperiod. Thus, the processor 1922 will initially show the AR objectoscillating (i.e., swinging) during a simulated seismic event, but willonly show the AR object falling if the preselected conditions are met.If the preselected conditions are not met, the AR object will continueswinging according to the primary manipulation type. In someembodiments, the processor 1922 may show the virtual representation ofthe object fall to the floor based on a highest slider value inoperation 2026. The virtual representation of the object canrealistically drop to the floor using standard physics for a fallingmotion of the virtual representation. When the virtual representation ofthe object reaches another object or a ground level, the virtualrepresentation can react with the other object or the ground level. Forinstance, a force could be applied to the other object based on thevirtual representation dropping to the floor. The virtual representationcan break or crack by the processor 1922 dividing the virtualrepresentation into two or more part virtual representations. Forinstance, a picture frame can reach the ground level and cause a crackthrough the picture frame creating two separate parts of the pictureframe. The two parts of the picture frame may operate independently fora remaining time of the simulation.

The virtual representation and background can be shaken independently.For example, a first picture frame can move independently from a secondpicture frame and a wall. The processor 1922 can create a set of parentcoordinate transforms in an application, such as Unity. The parenttransform can be an intermediate coordinate system between the worldcoordinate and children coordinates. The processor 1922 can set allsubmeshes from all regions that have a first face classification, suchas “floor”, to a first transform (also referred to as a parenttransform) and all submeshes from all regions that have a secondclassification, such as “wall”, to a second transform (also referred toas a child transform). The processor 1922 can set meshes that belong toa plane selected by a User as well as a plane with render cracks to athird transform. The processor 1922 can apply separate random shaking toeach of the parent transforms. Parent meshes ensure that a position ofthe sub meshes with respect to the parent transform remain constant.However, the position of the parent transform with respect to the worldis allowed to change. This allows all meshes assigned to a commonclassification to move together.

The shaking and falling of objects, such as picture frames, can beperformed by implementing Rigid body mechanics using a physic engine,such as a Unity Physics Engine. The selected wall plane and each pictureframe can be assigned rigid body functionalities, which allows theobjects to interact with each other in forms such as collisions, forces,constraints, and accelerations. The physics engine can provide functionsthat allow rigid bodies to relate to each other in a form ofconstraints. The physic engine can then iteratively solve multi-bodysimulation to give a next position of the rigid bodies given externalforces. The overall steps for this procedure can include (1) assigningrigid bodies to a picture frame and a wall mesh, and (2) adding arevolute hinge joint constraint between picture frame and wall mesh.

The processor 1922 can determine a damage amount 2812 for the scene 2702in operation 2914. The damage amount 2812 can be calculated as thesimulation is in progress and/or after the simulation of seismicactivity is completed. The damage amount 2812 can be calculated as apercentage, fraction, value, etc. The damage amount 2812 can beindividually calculated for each AR object. The processor 1922 canutilize specific effects applied to an object and an intensity of theeffect applied. For instance, an object that falls twice a distance as asimilar object could be determined to have more damage. The processor1922 can also determine a fragility of an object. For example, theprocessor 1922 can assign more damage to a vase falling an equaldistance to a book.

Although FIG. 29 illustrates an example method for an AR earthquakevisualization and assessment process 2900, various changes may be madeto FIG. 29 . For example, while shown as a series of steps, varioussteps in FIG. 29 may overlap, occur in parallel, occur in a differentorder, or occur any number of times.

It will be appreciated by those skilled in the art having the benefit ofthis disclosure that these methods and systems for display of anelectronic representation of physical effects and property damageresulting from a parametric natural disaster event provides an augmentedreality presentation of the personalized effect of an earthquake onactual objects around a user. It should be understood that the drawingsand detailed description herein are to be regarded in an illustrativerather than a restrictive manner, and are not intended to be limiting tothe particular forms and examples disclosed. On the contrary, includedare any further modifications, changes, rearrangements, substitutions,alternatives, design choices, and embodiments apparent to those ofordinary skill in the art, without departing from the spirit and scopehereof, as defined by the following claims. Thus, it is intended thatthe following claims be interpreted to embrace all such furthermodifications, changes, rearrangements, substitutions, alternatives,design choices, and embodiments.

What is claimed is:
 1. A method for displaying an augmented reality (AR)representation of physical effects and property damage resulting from aparametric earthquake event, comprising: scanning, using one or moresensors of a user device, a scene in proximity to a user; identifying abackground and objects in the scene; creating an AR background for thebackground of the scene and AR objects for the objects in the scene;receiving at least one seismic characteristic from the user of the userdevice; and displaying at least one seismic effect on the AR objects andthe AR background in the scene based on the at least one receivedseismic characteristic, and wherein displaying the at least one seismiceffect on the AR objects comprises: generating an AR crack on the ARbackground; and shading the AR crack to show three-dimensional parallaxbased on a point of view.
 2. A method in accordance with claim 1,wherein the background is a wall and the objects are furnishings.
 3. Amethod in accordance with claim 1, wherein displaying the at least oneseismic effect on the AR objects comprises: independently applying theat least one seismic effect on the AR objects and the AR background. 4.A method in accordance with claim 1, wherein, displaying the at leastone seismic effect on the AR objects, comprises: generating anoscillation on at least one of the AR objects.
 5. A method in accordancewith claim 1, wherein displaying the at least one seismic effect on theAR objects comprises: breaking an AR object into two or more AR partialobjects.
 6. A method in accordance with claim 5, wherein a size of eachpartial object is less than a size of the AR object.
 7. A method inaccordance with claim 1, further comprising: displaying a control for aseismic intensity as one of the at least one seismic characteristic. 8.A non-transitory computer readable medium, for displaying an augmentedreality (AR) representation of physical effects and property damageresulting from a parametric earthquake event, containing instructionsthat when executed cause a processor to: scan, using one or more sensorsof a user device, a scene in proximity to a user; identify a backgroundand objects in the scene; create an AR background for the background ofthe scene and AR objects for the objects in the scene; receive at leastone seismic characteristic from the user of the user device; and displayat least one seismic effect on the AR objects and the AR background inthe scene based on the at least one received seismic characteristic, andwherein the instructions when executed cause the processor to displaythe at least one seismic effect on the AR objects comprise instructionsthat when executed cause the processor to: generate an AR crack on theAR background; and shade the AR crack to show three-dimensional parallaxbased on a point of view.
 9. A non-transitory computer readable mediumin accordance with claim 8, wherein the background is a wall and theobjects are furnishings.
 10. A non-transitory computer readable mediumin accordance with claim 8, wherein the instructions when executed causethe processor to display the at least one seismic effect on the ARobjects comprise instructions that when executed cause the processor to:independently apply the at least one seismic effect on the AR objectsand the AR background.
 11. A non-transitory computer readable medium inaccordance with claim 8, wherein the instructions when executed causethe processor to display the at least one seismic effect on the ARobjects comprise instructions that when executed cause the processor to:generate an oscillation on at least one of the AR objects.
 12. Anon-transitory computer readable medium in accordance with claim 8,wherein the instructions when executed cause the processor to displaythe at least one seismic effect on the AR objects comprise instructionsthat when executed cause the processor to: break an AR object into twoor more AR partial objects.
 13. A non-transitory computer readablemedium in accordance with claim 12, wherein a cumulative size of eachpartial object is less than a size of the AR object.
 14. Anon-transitory computer readable medium in accordance with claim 8,wherein the instructions when executed further cause the processor to:display a control for a seismic intensity as one of the at least oneseismic characteristic.
 15. A method for displaying an augmented reality(AR) representation of physical effects and assessment of propertydamage resulting from a parametric earthquake event, comprising:scanning, using one or more sensors of a user device, a scene inproximity to a user; identifying a background and objects in the scene;counting the objects identified in the scene; creating an AR backgroundfor the background of the scene and AR objects for the objects in thescene; receiving at least one seismic characteristic from the user ofthe user device; displaying at least one seismic effect on the ARobjects and the AR background in the scene based on the at least onereceived seismic characteristic; determining a damage amount resultingfrom the parametric earthquake event based on a number of objectsidentified in the scene and the received seismic characteristic;displaying to the user an initial unit value for the background;allowing the user to provide a custom unit value for the background; andreplacing the initial unit value with the custom unit value as anassigned unit value used to determine a final unit value for estimatedstructure damage.
 16. A method displaying an augmented reality (AR)representation of physical effects and assessment of property damageresulting from a parametric earthquake event, comprising: scanning,using one or more sensors of a user device, a scene in proximity to auser; identifying a background and objects in the scene; counting theobjects identified in the scene; creating a first list of objectsidentified in a first scene and their respective assigned values;creating a second list of objects identified in a second scene and theirrespective assigned values; combining the first list of objects and thesecond list of objects into a consolidated object list including all theobject identified in the first scene and in the second scene and theirrespective assigned values; creating an AR background for the backgroundof the scene and AR objects for the objects in the scene; receiving atleast one seismic characteristic from the user of the user device;displaying at least one seismic effect on the AR objects and the ARbackground in the scene based on the at least one received seismiccharacteristic; and determining a damage amount resulting from theparametric earthquake event based on a number of objects identified inthe scene and the received seismic characteristic.
 17. A method inaccordance with claim 16, further comprising: using the consolidatedobject list to determine a consolidated object damage value for all theobjects identified in the first scene and in the second scene anddisplaying the consolidated object damage value to the user.